Oct 31 00:53:41.651965 ip-10-0-139-174 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 31 00:53:41.651978 ip-10-0-139-174 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 31 00:53:41.651984 ip-10-0-139-174 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 31 00:53:41.652226 ip-10-0-139-174 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 31 00:53:51.735684 ip-10-0-139-174 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 31 00:53:51.735701 ip-10-0-139-174 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b83be239d4b14f2499b751d2acc4a43e --
Oct 31 00:55:48.728459 ip-10-0-139-174 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 00:55:49.128817 ip-10-0-139-174 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:55:49.128817 ip-10-0-139-174 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 00:55:49.128817 ip-10-0-139-174 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:55:49.128817 ip-10-0-139-174 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:55:49.134501 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134418    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:55:49.134501 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134501    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:55:49.134501 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134505    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134508    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134511    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134514    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134517    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134527    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134530    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134533    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134535    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134538    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134540    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134543    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134545    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134550    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134555    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134559    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134561    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134565    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134567    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134570    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:55:49.134596 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134573    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134576    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134579    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134581    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134584    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134586    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134589    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134591    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134594    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134597    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134600    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134603    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134605    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134608    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134611    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134614    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134616    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134620    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134622    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134625    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:55:49.135056 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134627    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134630    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134632    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134634    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134637    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134639    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134642    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134644    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134647    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134649    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134652    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134654    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134656    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134659    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134662    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134664    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134668    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134672    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134675    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:55:49.135547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134678    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134680    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134683    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134686    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134689    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134691    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134694    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134696    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134698    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134701    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134704    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134707    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134710    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134714    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134716    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134719    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134721    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134724    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134727    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:55:49.136003 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134729    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134732    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134734    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134737    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134740    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134742    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134745    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134748    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134751    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134754    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134756    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134759    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134761    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134764    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134766    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134769    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134771    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134773    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134776    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:55:49.136469 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134778    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134780    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134783    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134786    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134789    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134792    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.134795    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135581    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135589    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135593    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135596    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135599    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135602    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135605    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135608    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135610    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135613    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135615    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135618    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135621    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:55:49.136963 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135623    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135626    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135628    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135631    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135634    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135637    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135640    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135642    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135645    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135647    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135650    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135653    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135655    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135658    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135660    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135663    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135666    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135668    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135670    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135673    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:55:49.137458 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135676    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135678    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135681    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135683    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135686    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135689    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135691    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135694    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135696    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135699    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135703    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135707    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135711    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135714    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135717    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135720    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135723    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135725    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135728    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:55:49.137934 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135730    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135733    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135735    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135737    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135740    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135742    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135745    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135747    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135750    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135752    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135754    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135757    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135759    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135763    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135765    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135768    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135771    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135773    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135776    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135778    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:55:49.138415 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135780    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135783    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135785    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135788    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135790    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135793    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135795    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135797    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135800    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135802    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135805    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135807    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135810    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135812    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135814    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135817    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135819    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135821    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135824    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:55:49.138885 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135826    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135830    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135834    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135838    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135841    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135843    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135846    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135849    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135852    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135855    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135858    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135861    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135864    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135867    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.135871    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136373    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136379    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136383    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136386    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:55:49.139353 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136388    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136391    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136394    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136398    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136402    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136405    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136408    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136410    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136413    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136415    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136418    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136420    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136423    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136425    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136428    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136431    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136434    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136436    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136439    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:55:49.139795 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136441    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136444    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136447    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136452    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136454    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136457    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136459    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136462    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136464    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136467    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136471    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136474    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136477    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136480    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136483    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136485    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136487    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136490    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136492    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136495    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:55:49.140255 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136497    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136500    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136502    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136505    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136507    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136510    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136513    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136515    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136518    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136521    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136524    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136527    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136529    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136531    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136534    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136538    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136541    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136543    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136546    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:55:49.140719 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136549    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136551    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136554    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136556    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136559    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136561    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136564    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136566    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136568    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136571    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136573    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136576    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136578    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136581    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136583    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136586    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136588    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136591    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136593    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:55:49.141224 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136596    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136598    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136601    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136604    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136606    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136608    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136611    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136613    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136615    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136620    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136623    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136625    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136629    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136631    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136634    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136636    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136639    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136641    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136644    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136646    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:55:49.141654 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136649    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136651    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136654    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136656    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.136658    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137609    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137620    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137630    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137635    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137639    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137643    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137647    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137651    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137654    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137659    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137663    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137666    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137669    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137672    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137675    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137678    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137681    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137685    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 00:55:49.142126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137689    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137691    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137695    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137698    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137701    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137705    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137707    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137710    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137714    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137717    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137720    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137722    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137725    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137728    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137732    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137735    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137738    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137741    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137744    2577 flags.go:64] FLAG: --enable-server="true"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137747    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137751    2577 flags.go:64] FLAG: --event-burst="100"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137754    2577 flags.go:64] FLAG: --event-qps="50"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137757    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137761    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 00:55:49.142669 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137764    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137768    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137771    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137774    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137777    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137780    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137783    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137786    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137789    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137792    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137795    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137798    2577 flags.go:64] FLAG: --feature-gates=""
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137802    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137805    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137808    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137811    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137814    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137817    2577 flags.go:64] FLAG: --help="false"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137820    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137823    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137826    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137829    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137832    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 00:55:49.143284 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137836    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137838    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137841    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137844    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137847    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137850    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137853    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137855    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137860    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137862    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137865    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137868    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137871    2577 flags.go:64] FLAG: --lock-file=""
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137874    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137877    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137880    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137885    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137888    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137891    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137894    2577 flags.go:64] FLAG: --logging-format="text"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137897    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137900    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137902    2577 flags.go:64] FLAG: --manifest-url=""
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137905    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137910    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 00:55:49.143824 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137913    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137917    2577 flags.go:64] FLAG: --max-pods="110"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137920    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137923    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137925    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137928    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137931    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137934    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137937    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137952    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137956    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137959    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137962    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137965    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137967    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137972    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137974    2577 flags.go:64] FLAG: --port="10250"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137977    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137980    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0fd33caf5398d3226"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137983    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137986    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137989    2577 flags.go:64] FLAG: --register-node="true"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137992    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137996    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.137999    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138001    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 00:55:49.144455 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138005    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138009    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138011    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138015    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138018    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138021    2577 flags.go:64] FLAG: --runonce="false"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138025    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138028    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138031    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138033    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138036    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138039    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138042    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138045    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138048    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138051    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138054    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138056    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138059    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138062    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138065    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138071    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138074    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138077    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138080    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 31 00:55:49.145104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138083    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138085    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138088    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138091    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138094    2577 flags.go:64] FLAG: --v="2"
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138101    2577 flags.go:64] FLAG: --version="false"
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138105    2577 flags.go:64] FLAG: --vmodule=""
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138109    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138113    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138258    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138262    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138264    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138267    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138272    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138275    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138277    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138280    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138283    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138285    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138288    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138290    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:55:49.145702 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138293    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138295    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138298    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138301    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138303    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138306    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138308    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138312    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138315    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138318    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138320    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138323    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138325    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138327    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138330    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138332    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138335    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138337    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138340    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138342    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:55:49.146212 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138345    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138348    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138350    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138352    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138356    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138359    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138361    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138363    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138366    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138368    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138371    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138373    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138376    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138378    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138381    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138383    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138386    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138388    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138391    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138394    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:55:49.146687 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138397    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138401    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138404    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138408    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138410    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138413    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138416    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138419    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138421    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138424    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138426    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138429    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138431    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138434    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138437    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138440    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138445    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138448    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138451    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:55:49.147162 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138453    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138456    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138459    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138461    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138464    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138466    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138468    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138471    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138474    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138476    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138478    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138481    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138485    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138487    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138490    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138492    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138495    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138497    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138500    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:55:49.147649 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138502    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138505    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138507    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138509    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138512    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138514    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138517    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138520    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138523    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138526    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138530    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138532    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138535    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138538    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138540    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:55:49.148093 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138543    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.138569    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138602    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138607    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138609    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138612    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138615    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138617    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138620    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138622    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138626    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138629    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138631    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138633    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138636    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138639    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:55:49.148463 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138641    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138643    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138646    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138648    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138651    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138653    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138656    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138658    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138660    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138663    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138666    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138668    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138673    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138675    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138677    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138680    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138682    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138685    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138687    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138689    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:55:49.148832 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138692    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138694    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138697    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138699    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138702    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138704    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138708    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138710    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138712    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138715    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138717    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138721    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138725    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138728    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138730    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138733    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138735    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138738    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138740    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:55:49.149547 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138742    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138746    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138748    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138750    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138753    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138757    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138760    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138764    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138767    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138770    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138773    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138775    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138778    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138780    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138783    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138785    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138788    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138790    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138794    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:55:49.150256 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138798    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138800    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138803    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138805    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138808    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138810    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138812    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138815    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138817    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138820    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138822    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138824    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138827    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138829    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138831    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138834    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138836    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138839    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138843    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138845    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:55:49.150746 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138848    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138850    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138853    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138856    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138858    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138861    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138864    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138866    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138868    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138871    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138874    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138876    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138880    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.138882    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.145840    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.145855    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.146458    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.146464    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146521    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:55:49.151247 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146527    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146530    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146534    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146537    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146539    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146542    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146546    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146550    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146553    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146555    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146558    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146561    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146564    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146567    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146570    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146573    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146576    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146578    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146580    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146583    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:55:49.151693 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146586    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146589    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146591    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146594    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146596    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146599    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146602    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146605    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146609    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146611    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146614    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146616    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146619    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146621    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146624    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146626    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146628    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146631    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146633    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146635    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:55:49.152166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146638    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146640    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146643    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146645    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146648    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146652    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146655    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146658    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146660    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146663    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146665    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146668    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146670    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146673    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146675    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146677    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146680    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146683    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146685    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:55:49.152717 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146688    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146691    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146694    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146697    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146700    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146703    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146705    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146707    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146710    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146713    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146715    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146718    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146720    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146722    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146725    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146728    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146730    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146732    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146735    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146737    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:55:49.153166 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146740    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146742    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146745    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146747    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146749    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146752    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146754    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146756    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146759    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146762    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146764    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146767    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146770    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146772    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146774    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146778    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146780    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146782    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146785    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146787    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:55:49.153635 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146789    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146792    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146794    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146796    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146799    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146801    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.146829    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146965    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146969    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146972    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146974    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146977    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146979    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146982    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146985    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146987    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:55:49.154124 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146990    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146992    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146995    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.146997    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147000    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147002    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147005    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147007    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147010    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147012    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147015    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147017    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147020    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147022    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147024    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147027    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147029    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147032    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147034    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147036    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:55:49.154518 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147039    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147041    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147043    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147046    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147048    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147051    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147053    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147055    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147058    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147060    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147063    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147065    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147067    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147070    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147072    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147074    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147077    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147079    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147082    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147084    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:55:49.154984 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147086    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147089    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147091    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147093    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147096    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147099    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147102    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147104    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147106    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147109    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147111    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147113    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147116    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147118    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147120    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147123    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147125    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147127    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147130    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147132    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:55:49.155590 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147135    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147137    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147139    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147142    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147144    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147147    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147149    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147152    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147154    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147156    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147159    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147162    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147165    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147167    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147183    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147186    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147188    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147191    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147194    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:55:49.156061 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147196    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147200    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147205    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147209    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147212    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147215    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147218    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147221    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147223    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147225    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147228    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147230    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147232    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147235    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147237    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147239    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147242    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:55:49.156514 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147245    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.147267    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147298    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147302    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147306    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147309    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147313    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147316    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147318    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147321    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147323    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147325    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147328    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147330    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147333    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:55:49.156904 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147337    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147339    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147342    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147345    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147347    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147349    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147352    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147355    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147359    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147362    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147364    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147367    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147370    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147372    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147375    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147377    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147380    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147383    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147385    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147388    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:55:49.157282 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147390    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147393    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147395    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147397    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147400    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147402    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147405    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147407    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147409    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147412    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147415    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147417    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147419    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147422    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147427    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147429    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147432    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147434    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147437    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147439    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:55:49.157815 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147443    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147445    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147448    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147450    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147453    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147455    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147458    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147460    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147463    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147465    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147468    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147470    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147472    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147475    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147477    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147479    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147482    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147484    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147487    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:55:49.158316 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147489    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147492    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147494    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147497    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147499    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147501    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147504    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147508    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147510    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147513    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147515    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147518    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147520    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147524    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147527    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147529    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147532    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147534    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147536    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147539    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:55:49.158910 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147541    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147544    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147546    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147548    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147551    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147553    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147556    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147558    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147561    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147564    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147566    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147568    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147571    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:49.147573    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.147624    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.150300    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.151634    2577 server.go:1014] "Starting client certificate rotation"
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.151744    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:55:49.159420 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.151777    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:55:49.159831 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.153692    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:55:49.159831 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.157502    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:55:49.167062 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.167048    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 31 00:55:49.172308 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.172293    2577 log.go:25] "Validated CRI v1 image API"
Oct 31 00:55:49.173111 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.173089    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 00:55:49.175718 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.175699    2577 fs.go:135] Filesystem UUIDs: map[07052935-016b-4273-8522-16f400d4650f:/dev/nvme0n1p3 3826533a-182a-4dab-8b4b-22e346293988:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 31 00:55:49.175792 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.175717    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 00:55:49.180869 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.180745    2577 manager.go:217] Machine: {Timestamp:2026-10-31 00:55:49.179334109 +0000 UTC m=+0.341053760 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102608 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24c640e66eb85dc6d742837fe9cd36 SystemUUID:ec24c640-e66e-b85d-c6d7-42837fe9cd36 BootID:b83be239-d4b1-4f24-99b7-51d2acc4a43e Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:24:40:e3:23:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:24:40:e3:23:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:7f:4d:94:40:aa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 00:55:49.180869 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.180870    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 00:55:49.180975 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.180950    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 00:55:49.184166 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.184147    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 00:55:49.184321 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.184169    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-174.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 00:55:49.184364 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.184324    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 00:55:49.184364 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.184332    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 00:55:49.184364 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.184362    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:55:49.184442 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.184379    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:55:49.185635 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.185625    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 00:55:49.190028 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.190016    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 00:55:49.190143 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.190134    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 00:55:49.190967 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.190957    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 00:55:49.191016 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.190974    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 00:55:49.191016 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.190994    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 00:55:49.191016 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.191006    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 31 00:55:49.191016 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.191015    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 00:55:49.191441 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.191427    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:55:49.193815 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.193797    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 00:55:49.201923 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.201907    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:55:49.203449 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.203437    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 00:55:49.203501 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.203467    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 00:55:49.203638 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.203628    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 00:55:49.203682 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.203645    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 00:55:49.203682 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.203651    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 00:55:49.203682 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.203657    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 00:55:49.203682 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.203662    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 00:55:49.203682 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.203669    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 00:55:49.204731 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.204722    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 00:55:49.204761 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.204731    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 00:55:49.204761 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.204739    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 00:55:49.204761 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.204745    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 00:55:49.204761 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.204754    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 00:55:49.204761 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.204762    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 00:55:49.205522 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.205512    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 00:55:49.205557 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.205525    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 00:55:49.209274 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.209261    2577 server.go:1263] "Started kubelet"
Oct 31 00:55:49.209374 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.209325    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 00:55:49.209468 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.209350    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 00:55:49.209468 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.209407    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 00:55:49.209998 ip-10-0-139-174 systemd[1]: Started Kubernetes Kubelet.
Oct 31 00:55:49.210762 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.210701    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 00:55:49.212650 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.212633    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 00:55:49.216170 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.216148    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 00:55:49.217837 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.217816    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 00:55:49.219423 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.219409    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 00:55:49.219527 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.219502    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-174.us-west-1.compute.internal\" not found"
Oct 31 00:55:49.220136 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.220115    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 00:55:49.220272 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.220126    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 00:55:49.220355 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.220327    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 00:55:49.220355 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.220352    2577 factory.go:55] Registering systemd factory
Oct 31 00:55:49.220459 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.220367    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 31 00:55:49.220459 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.220430    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 00:55:49.220459 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.220438    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 00:55:49.220602 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.220597    2577 factory.go:153] Registering CRI-O factory
Oct 31 00:55:49.220645 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.220604    2577 factory.go:223] Registration of the crio container factory successfully
Oct 31 00:55:49.220675 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.220661    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 00:55:49.220704 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.220681    2577 factory.go:103] Registering Raw factory
Oct 31 00:55:49.220704 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.220699    2577 manager.go:1199] Started watching for new ooms in manager
Oct 31 00:55:49.221896 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.221880    2577 manager.go:319] Starting recovery of all containers
Oct 31 00:55:49.233934 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.233915    2577 manager.go:324] Recovery completed
Oct 31 00:55:49.239055 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.239040    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:55:49.240651 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.240633    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:55:49.240699 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.240666    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:55:49.240699 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.240677    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:55:49.241241 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.241227    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 31 00:55:49.241241 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.241238    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 00:55:49.241394 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.241245    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:55:49.241394 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.241257    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 00:55:49.242114 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.242099    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:55:49.242199 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.242125    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:55:49.242199 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.242134    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:55:49.243418 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.243404    2577 policy_none.go:50] "Start"
Oct 31 00:55:49.243418 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.243421    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 00:55:49.243561 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.243434    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 00:55:49.245412 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.245400    2577 policy_none.go:44] "Start"
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.284717    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.287086    2577 manager.go:341] "Starting Device Plugin manager"
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.287288    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.287301    2577 server.go:87] "Starting device plugin registration server"
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.287537    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.287549    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.287642    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.287735    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.287746    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.288266    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.288306    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-174.us-west-1.compute.internal\" not found"
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.297950    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.299354    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.299375    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.299394    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 00:55:49.305151 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.299431    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 00:55:49.350251 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.350224    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-174.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 00:55:49.351344 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.350224    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-174.us-west-1.compute.internal.18e37735b6e82e1d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-174.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-174.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 00:55:49.209230877 +0000 UTC m=+0.370950529,LastTimestamp:2026-10-31 00:55:49.209230877 +0000 UTC m=+0.370950529,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-174.us-west-1.compute.internal,}"
Oct 31 00:55:49.388492 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.388442    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:55:49.389342 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.389327    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:55:49.389391 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.389374    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:55:49.389391 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.389387    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:55:49.389453 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.389413    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.399573 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.399552    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal"]
Oct 31 00:55:49.399686 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.399630    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:55:49.399726 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.399688    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-174.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 00:55:49.400399 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.400375    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-174.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 00:55:49.400535 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.400424    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 00:55:49.400535 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.400444    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:55:49.400535 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.400473    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:55:49.400535 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.400483    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:55:49.402744 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.402732    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:55:49.402895 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.402882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.402950 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.402912    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:55:49.403451 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.403437    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:55:49.403556 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.403448    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:55:49.403556 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.403460    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:55:49.403556 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.403470    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:55:49.403556 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.403471    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:55:49.403556 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.403482    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:55:49.405620 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.405605    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.405719 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.405635    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:55:49.406321 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.406305    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:55:49.406397 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.406337    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:55:49.406397 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.406347    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:55:49.434576 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.434560    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-174.us-west-1.compute.internal\" not found" node="ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.436541 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.436526    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-174.us-west-1.compute.internal\" not found" node="ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.477819 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.477800    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 00:55:49.477901 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.477869    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dr5v8"
Oct 31 00:55:49.503633 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.503604    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 00:55:49.503706 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.503672    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 31 00:55:49.503769 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.503697    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-174.us-west-1.compute.internal.18e37735b8c79b14  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-174.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-174.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-174.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 00:55:49.240650516 +0000 UTC m=+0.402370168,LastTimestamp:2026-10-31 00:55:49.240650516 +0000 UTC m=+0.402370168,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-174.us-west-1.compute.internal,}"
Oct 31 00:55:49.526126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.526088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fc072a2ae8e85ec3464a94d2bac4401-config\") pod \"kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"7fc072a2ae8e85ec3464a94d2bac4401\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.526126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.526129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52cc4e07643ef7c5669e5939f9f4241e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"52cc4e07643ef7c5669e5939f9f4241e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.526283 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.526144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52cc4e07643ef7c5669e5939f9f4241e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"52cc4e07643ef7c5669e5939f9f4241e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.557326 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.557267    2577 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-139-174.us-west-1.compute.internal\" not found" node="ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.626534 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.626509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52cc4e07643ef7c5669e5939f9f4241e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"52cc4e07643ef7c5669e5939f9f4241e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.626534 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.626536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fc072a2ae8e85ec3464a94d2bac4401-config\") pod \"kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"7fc072a2ae8e85ec3464a94d2bac4401\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.626683 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.626552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52cc4e07643ef7c5669e5939f9f4241e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"52cc4e07643ef7c5669e5939f9f4241e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.626683 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.626609    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52cc4e07643ef7c5669e5939f9f4241e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"52cc4e07643ef7c5669e5939f9f4241e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.626683 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.626619    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fc072a2ae8e85ec3464a94d2bac4401-config\") pod \"kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"7fc072a2ae8e85ec3464a94d2bac4401\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.626683 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.626610    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52cc4e07643ef7c5669e5939f9f4241e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"52cc4e07643ef7c5669e5939f9f4241e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.704316 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.704245    2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-174.us-west-1.compute.internal\" not found" node="ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.738752 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.738731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.741885 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.741867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.758380 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.758365    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:55:49.759322 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.759303    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:55:49.759385 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.759344    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:55:49.759385 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.759356    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:55:49.759385 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.759382    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.814954 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:49.814934    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:49.815021 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.814961    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-174.us-west-1.compute.internal\": node \"ip-10-0-139-174.us-west-1.compute.internal\" not found"
Oct 31 00:55:49.937949 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:49.937911    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-174.us-west-1.compute.internal\" not found"
Oct 31 00:55:50.038620 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:50.038557    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-174.us-west-1.compute.internal\" not found"
Oct 31 00:55:50.138867 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:50.138833    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-174.us-west-1.compute.internal\" not found"
Oct 31 00:55:50.151027 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.150990    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 00:55:50.151290 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:50.151214    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a747793b6d5cd47a4aa82ac69821d294-e9c03ac64371c6c0.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-174.us-west-1.compute.internal.18e37735b8c79b14\": read tcp 10.0.139.174:36402->3.135.74.141:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-174.us-west-1.compute.internal.18e37735b8c79b14  default   5035 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-174.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-174.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-174.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 00:55:49 +0000 UTC,LastTimestamp:2026-10-31 00:55:49.400456199 +0000 UTC m=+0.562175851,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-174.us-west-1.compute.internal,}"
Oct 31 00:55:50.238951 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:50.238923    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-174.us-west-1.compute.internal\" not found"
Oct 31 00:55:50.339051 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:50.338982    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-174.us-west-1.compute.internal\" not found"
Oct 31 00:55:50.439516 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:50.439486    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-174.us-west-1.compute.internal\" not found"
Oct 31 00:55:50.483304 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.483258    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:50:50 +0000 UTC" deadline="2028-04-16 11:41:16.644777157 +0000 UTC"
Oct 31 00:55:50.483304 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.483299    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12802h45m26.161482582s"
Oct 31 00:55:50.503489 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.503472    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:55:50.504478 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.504460    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:55:50.504590 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.504495    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:55:50.504590 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.504511    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:55:50.517586 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.517564    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:55:50.519836 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.519816    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:50.568051 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.568025    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:55:50.613731 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.613673    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:55:50.614801 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.614785    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:55:50.671660 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.671636    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:55:50.813070 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:50.813042    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52cc4e07643ef7c5669e5939f9f4241e.slice/crio-c4459dedf25045e99b03c872e7883f37e2c7ddef022171c54d199e6d20e75bab WatchSource:0}: Error finding container c4459dedf25045e99b03c872e7883f37e2c7ddef022171c54d199e6d20e75bab: Status 404 returned error can't find the container with id c4459dedf25045e99b03c872e7883f37e2c7ddef022171c54d199e6d20e75bab
Oct 31 00:55:50.813560 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:50.813537    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc072a2ae8e85ec3464a94d2bac4401.slice/crio-aaf8df7b834dd3e771643f1bc0143c52b7f25f9bcfb196d2c85069811d3ca94f WatchSource:0}: Error finding container aaf8df7b834dd3e771643f1bc0143c52b7f25f9bcfb196d2c85069811d3ca94f: Status 404 returned error can't find the container with id aaf8df7b834dd3e771643f1bc0143c52b7f25f9bcfb196d2c85069811d3ca94f
Oct 31 00:55:50.819873 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.819850    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:55:50.857225 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.857204    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:55:50.907557 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:50.907492    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:55:51.193158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.193073    2577 apiserver.go:52] "Watching apiserver"
Oct 31 00:55:51.219186 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.219156    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 00:55:51.304170 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.304138    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 00:55:51.304622 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.304531    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal" event={"ID":"7fc072a2ae8e85ec3464a94d2bac4401","Type":"ContainerStarted","Data":"aaf8df7b834dd3e771643f1bc0143c52b7f25f9bcfb196d2c85069811d3ca94f"}
Oct 31 00:55:51.304709 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.304653    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-7rmmh","kube-system/global-pull-secret-syncer-jdmsp","kube-system/konnectivity-agent-jww65","openshift-image-registry/node-ca-lthm8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj","openshift-multus/multus-additional-cni-plugins-g2k97","openshift-network-diagnostics/network-check-target-qhvwh","openshift-network-operator/iptables-alerter-djz6l","openshift-ovn-kubernetes/ovnkube-node-46s2r","openshift-cluster-node-tuning-operator/tuned-n2x8d","openshift-dns/node-resolver-gsv6r","openshift-multus/network-metrics-daemon-rlr9b","kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"]
Oct 31 00:55:51.308941 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.308917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.311260 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.311230    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:51.311348 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:51.311307    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:55:51.313600 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.313579    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:55:51.315864 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.315848    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:55:51.318275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.318238    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.321297 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.321277    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.323775 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.323759    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:55:51.323856 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:51.323815    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:55:51.326462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.326439    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:55:51.328914 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.328893    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.331402 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.331384    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.333777 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.333761    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:55:51.336324 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.336294    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal" event={"ID":"52cc4e07643ef7c5669e5939f9f4241e","Type":"ContainerStarted","Data":"c4459dedf25045e99b03c872e7883f37e2c7ddef022171c54d199e6d20e75bab"}
Oct 31 00:55:51.336426 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.336405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:55:51.336537 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:51.336517    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:55:51.337351 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rz6k\" (UniqueName: \"kubernetes.io/projected/27c82039-550c-4fe8-96c7-dfd8990440e4-kube-api-access-7rz6k\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.337446 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cphjb\" (UniqueName: \"kubernetes.io/projected/f21550fa-496b-4806-abd8-7f0441da9256-kube-api-access-cphjb\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.337446 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-cnibin\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.337446 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.337446 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-system-cni-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.337695 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-cni-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.337695 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-conf-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.337695 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-socket-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.337695 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.337695 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h86lw\" (UniqueName: \"kubernetes.io/projected/12564994-4d47-4401-bc99-a4755b7ed33f-kube-api-access-h86lw\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.337695 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-system-cni-dir\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.337695 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.337695 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.337990 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-hostroot\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.337990 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/12564994-4d47-4401-bc99-a4755b7ed33f-multus-daemon-config\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.337990 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-etc-kubernetes\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.337990 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:51.337990 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-k8s-cni-cncf-io\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.337990 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b893def-2fc6-4350-a01f-6ee4969c48bd-agent-certs\") pod \"konnectivity-agent-jww65\" (UID: \"7b893def-2fc6-4350-a01f-6ee4969c48bd\") " pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:55:51.337990 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t522b\" (UniqueName: \"kubernetes.io/projected/e4311d66-bb16-4a76-a021-527f4c1395f4-kube-api-access-t522b\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:55:51.337990 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-registration-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.337990 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.337990 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.337990 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.337976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12564994-4d47-4401-bc99-a4755b7ed33f-cni-binary-copy\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.338454 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.338017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-netns\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.338454 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.338033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-kubelet\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.338454 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.338047    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-multus-certs\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.338454 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.338062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4311d66-bb16-4a76-a021-527f4c1395f4-host\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:55:51.338454 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.338086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4311d66-bb16-4a76-a021-527f4c1395f4-serviceca\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:55:51.338454 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.338115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-cnibin\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.338454 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.338134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-socket-dir-parent\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.338454 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.338154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0cade88-4da4-467c-9656-cce6afe1dd62-dbus\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:51.338454 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.338168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-device-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.338454 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.338237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-cni-bin\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.338454 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.338271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-cni-multus\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.338454 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.338299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0cade88-4da4-467c-9656-cce6afe1dd62-kubelet-config\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:51.338454 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.338340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b893def-2fc6-4350-a01f-6ee4969c48bd-konnectivity-ca\") pod \"konnectivity-agent-jww65\" (UID: \"7b893def-2fc6-4350-a01f-6ee4969c48bd\") " pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:55:51.338454 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.338391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-sys-fs\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.365365 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.365342    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 00:55:51.365365 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.365342    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 00:55:51.365524 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.365343    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 00:55:51.365524 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.365381    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-98n4j\""
Oct 31 00:55:51.365524 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.365412    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 00:55:51.369207 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.369188    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xr8df\""
Oct 31 00:55:51.369207 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.369205    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 00:55:51.369342 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.369221    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 00:55:51.370971 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.370950    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gr6qq\""
Oct 31 00:55:51.406365 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.406336    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:55:51.406365 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.406352    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 00:55:51.406581 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.406406    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 00:55:51.406581 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.406436    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 00:55:51.406581 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.406448    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wfh75\""
Oct 31 00:55:51.406581 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.406455    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 00:55:51.406581 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.406462    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 00:55:51.406581 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.406473    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6sjwh\""
Oct 31 00:55:51.406581 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.406451    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 00:55:51.406581 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.406464    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 00:55:51.406581 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.406436    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 00:55:51.406581 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.406508    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 00:55:51.406871 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.406652    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 00:55:51.412516 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.412499    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 00:55:51.412649 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.412637    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 00:55:51.412891 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.412855    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:55:51.412891 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.412886    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 00:55:51.413099 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.413087    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 00:55:51.413138 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.413087    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 00:55:51.418673 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.418657    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 00:55:51.418796 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.418779    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lhvrj\""
Oct 31 00:55:51.418852 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.418786    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 00:55:51.418852 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.418803    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fksdd\""
Oct 31 00:55:51.418852 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.418824    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 00:55:51.418976 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.418865    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ndfs4\""
Oct 31 00:55:51.418976 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.418872    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g8n5b\""
Oct 31 00:55:51.419123 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.419097    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 00:55:51.419247 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.419125    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 00:55:51.419247 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.419145    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-skqz4"
Oct 31 00:55:51.421904 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.421890    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 00:55:51.439604 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.439585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-conf-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.439673 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.439609    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-socket-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.439673 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.439629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-lib-modules\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.439673 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.439645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-systemd-units\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.439673 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.439663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfxdb\" (UniqueName: \"kubernetes.io/projected/68af8611-a641-4663-b13b-8f015107e82f-kube-api-access-kfxdb\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.439811 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.439729    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-conf-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.439811 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.439753    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-socket-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.439811 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.439762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.439811 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.439798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-modprobe-d\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.439955 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.439902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-hosts-file\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:55:51.439986 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.439953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-tmp-dir\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:55:51.440028 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.439983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-slash\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.440028 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-host\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.440098 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-system-cni-dir\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.440098 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.440164 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.440164 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440124    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-system-cni-dir\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.440164 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-etc-kubernetes\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.440305 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:51.440305 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440213    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-etc-kubernetes\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.440305 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440238    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.440305 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t522b\" (UniqueName: \"kubernetes.io/projected/e4311d66-bb16-4a76-a021-527f4c1395f4-kube-api-access-t522b\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:55:51.440305 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysconfig\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.440305 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-k8s-cni-cncf-io\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.440511 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-registration-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.440511 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr7vl\" (UniqueName: \"kubernetes.io/projected/ac61d127-75ba-4f73-bcc6-7d69bf320a55-kube-api-access-vr7vl\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:55:51.440511 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440343    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-k8s-cni-cncf-io\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.440511 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.440511 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440367    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.440511 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440387    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-registration-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.440511 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-cni-netd\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.440511 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.440784 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440680    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.440830 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:51.440814    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:55:51.440883 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-env-overrides\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.440933 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:51.440902    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret podName:e0cade88-4da4-467c-9656-cce6afe1dd62 nodeName:}" failed. No retries permitted until 2026-10-31 00:55:51.940886521 +0000 UTC m=+3.102606163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret") pod "global-pull-secret-syncer-jdmsp" (UID: "e0cade88-4da4-467c-9656-cce6afe1dd62") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:55:51.440933 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12564994-4d47-4401-bc99-a4755b7ed33f-cni-binary-copy\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441014 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-netns\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441014 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440948    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-kubelet\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441014 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-multus-certs\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441014 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.440984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4311d66-bb16-4a76-a021-527f4c1395f4-host\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:55:51.441178 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-cnibin\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.441178 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441020    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-netns\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441178 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441030    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-kubelet\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441178 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0cade88-4da4-467c-9656-cce6afe1dd62-dbus\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:51.441178 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441047    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-multus-certs\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441178 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441070    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-cnibin\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.441178 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-device-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.441178 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441092    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4311d66-bb16-4a76-a021-527f4c1395f4-host\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:55:51.441178 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysctl-d\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.441178 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441113    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-device-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.441178 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-var-lib-kubelet\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.441178 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9mcc\" (UniqueName: \"kubernetes.io/projected/9bd7a453-d0b1-4db7-8dfa-247a72467481-kube-api-access-q9mcc\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-ovnkube-script-lib\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-cni-bin\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441281    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-cni-bin\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-cni-multus\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0cade88-4da4-467c-9656-cce6afe1dd62-kubelet-config\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441321    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0cade88-4da4-467c-9656-cce6afe1dd62-dbus\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b893def-2fc6-4350-a01f-6ee4969c48bd-konnectivity-ca\") pod \"konnectivity-agent-jww65\" (UID: \"7b893def-2fc6-4350-a01f-6ee4969c48bd\") " pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rz6k\" (UniqueName: \"kubernetes.io/projected/27c82039-550c-4fe8-96c7-dfd8990440e4-kube-api-access-7rz6k\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441374    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12564994-4d47-4401-bc99-a4755b7ed33f-cni-binary-copy\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441390    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0cade88-4da4-467c-9656-cce6afe1dd62-kubelet-config\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-tuned\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441523    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-cni-multus\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cphjb\" (UniqueName: \"kubernetes.io/projected/f21550fa-496b-4806-abd8-7f0441da9256-kube-api-access-cphjb\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-cnibin\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-system-cni-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441649    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-cnibin\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-run\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.441783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441671    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-system-cni-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-sys\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-var-lib-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-542pt\" (UniqueName: \"kubernetes.io/projected/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-kube-api-access-542pt\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86lw\" (UniqueName: \"kubernetes.io/projected/12564994-4d47-4401-bc99-a4755b7ed33f-kube-api-access-h86lw\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-kubernetes\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441853    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b893def-2fc6-4350-a01f-6ee4969c48bd-konnectivity-ca\") pod \"konnectivity-agent-jww65\" (UID: \"7b893def-2fc6-4350-a01f-6ee4969c48bd\") " pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac61d127-75ba-4f73-bcc6-7d69bf320a55-host-slash\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-cni-bin\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-hostroot\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/12564994-4d47-4401-bc99-a4755b7ed33f-multus-daemon-config\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b893def-2fc6-4350-a01f-6ee4969c48bd-agent-certs\") pod \"konnectivity-agent-jww65\" (UID: \"7b893def-2fc6-4350-a01f-6ee4969c48bd\") " pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.441996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442002    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-hostroot\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pdzc\" (UniqueName: \"kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc\") pod \"network-check-target-qhvwh\" (UID: \"74673206-8b68-4414-8ac0-a63577cea44a\") " pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-kubelet\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-ovn\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.442597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-node-log\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442119    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-log-socket\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4311d66-bb16-4a76-a021-527f4c1395f4-serviceca\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysctl-conf\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-systemd\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442390    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442442    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/12564994-4d47-4401-bc99-a4755b7ed33f-multus-daemon-config\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bd7a453-d0b1-4db7-8dfa-247a72467481-tmp\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-socket-dir-parent\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-systemd\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-etc-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-ovnkube-config\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68af8611-a641-4663-b13b-8f015107e82f-ovn-node-metrics-cert\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-sys-fs\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.443275 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442649    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-socket-dir-parent\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.443768 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vwtr\" (UniqueName: \"kubernetes.io/projected/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-kube-api-access-2vwtr\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:55:51.443768 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-run-ovn-kubernetes\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.443768 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442692    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-sys-fs\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.443768 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.443768 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac61d127-75ba-4f73-bcc6-7d69bf320a55-iptables-alerter-script\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:55:51.443768 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442751    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.443768 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442754    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4311d66-bb16-4a76-a021-527f4c1395f4-serviceca\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:55:51.443768 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-run-netns\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.443768 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442760    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.443768 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-cni-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.443768 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.442863    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-cni-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.446113 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.446083    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b893def-2fc6-4350-a01f-6ee4969c48bd-agent-certs\") pod \"konnectivity-agent-jww65\" (UID: \"7b893def-2fc6-4350-a01f-6ee4969c48bd\") " pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:55:51.503226 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.503194    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rz6k\" (UniqueName: \"kubernetes.io/projected/27c82039-550c-4fe8-96c7-dfd8990440e4-kube-api-access-7rz6k\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.503226 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.503220    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86lw\" (UniqueName: \"kubernetes.io/projected/12564994-4d47-4401-bc99-a4755b7ed33f-kube-api-access-h86lw\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.503370 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.503233    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cphjb\" (UniqueName: \"kubernetes.io/projected/f21550fa-496b-4806-abd8-7f0441da9256-kube-api-access-cphjb\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.503370 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.503192    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t522b\" (UniqueName: \"kubernetes.io/projected/e4311d66-bb16-4a76-a021-527f4c1395f4-kube-api-access-t522b\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:55:51.543044 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-kubernetes\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.543044 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac61d127-75ba-4f73-bcc6-7d69bf320a55-host-slash\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:55:51.543217 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-cni-bin\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.543217 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdzc\" (UniqueName: \"kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc\") pod \"network-check-target-qhvwh\" (UID: \"74673206-8b68-4414-8ac0-a63577cea44a\") " pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:55:51.543217 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-kubelet\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.543217 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-ovn\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.543217 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543151    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-cni-bin\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.543217 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-node-log\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.543217 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543151    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-kubernetes\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.543217 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-log-socket\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543116    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac61d127-75ba-4f73-bcc6-7d69bf320a55-host-slash\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543230    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-kubelet\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543246    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-node-log\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543264    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-ovn\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysctl-conf\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543285    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-log-socket\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-systemd\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:51.543305    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bd7a453-d0b1-4db7-8dfa-247a72467481-tmp\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-systemd\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543335    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-systemd\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543353    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-systemd\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:51.543360    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs podName:9d67aaf2-9211-4f6c-84d8-99aae85c28eb nodeName:}" failed. No retries permitted until 2026-10-31 00:55:52.043342285 +0000 UTC m=+3.205061943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs") pod "network-metrics-daemon-rlr9b" (UID: "9d67aaf2-9211-4f6c-84d8-99aae85c28eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-etc-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543431    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-etc-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.543619 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543429    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysctl-conf\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-ovnkube-config\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68af8611-a641-4663-b13b-8f015107e82f-ovn-node-metrics-cert\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vwtr\" (UniqueName: \"kubernetes.io/projected/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-kube-api-access-2vwtr\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-run-ovn-kubernetes\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac61d127-75ba-4f73-bcc6-7d69bf320a55-iptables-alerter-script\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543623    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-run-netns\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-lib-modules\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543655    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-run-ovn-kubernetes\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-systemd-units\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfxdb\" (UniqueName: \"kubernetes.io/projected/68af8611-a641-4663-b13b-8f015107e82f-kube-api-access-kfxdb\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543728    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-systemd-units\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-modprobe-d\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543778    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-run-netns\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-hosts-file\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-tmp-dir\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-slash\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.544462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-host\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543909    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-lib-modules\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543921    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-hosts-file\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysconfig\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7vl\" (UniqueName: \"kubernetes.io/projected/ac61d127-75ba-4f73-bcc6-7d69bf320a55-kube-api-access-vr7vl\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543993    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-modprobe-d\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.543977    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-slash\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544042    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-host\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-cni-netd\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544206    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544208    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-ovnkube-config\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544235    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac61d127-75ba-4f73-bcc6-7d69bf320a55-iptables-alerter-script\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544256    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544260    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-tmp-dir\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544266    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-cni-netd\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.545158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544293    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysconfig\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-env-overrides\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysctl-d\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-var-lib-kubelet\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9mcc\" (UniqueName: \"kubernetes.io/projected/9bd7a453-d0b1-4db7-8dfa-247a72467481-kube-api-access-q9mcc\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-ovnkube-script-lib\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-tuned\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544482    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-var-lib-kubelet\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-run\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544526    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysctl-d\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-sys\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-var-lib-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544586    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-sys\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-542pt\" (UniqueName: \"kubernetes.io/projected/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-kube-api-access-542pt\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544706    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-env-overrides\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544771    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-var-lib-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544861    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-run\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.545667 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.544959    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-ovnkube-script-lib\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.546276 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.546191    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68af8611-a641-4663-b13b-8f015107e82f-ovn-node-metrics-cert\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.546276 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.546230    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bd7a453-d0b1-4db7-8dfa-247a72467481-tmp\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.546760 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.546743    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-tuned\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.602919 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:51.602894    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:55:51.602919 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:51.602918    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:55:51.603117 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:51.602929    2577 projected.go:196] Error preparing data for projected volume kube-api-access-5pdzc for pod openshift-network-diagnostics/network-check-target-qhvwh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:55:51.603117 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:51.602991    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc podName:74673206-8b68-4414-8ac0-a63577cea44a nodeName:}" failed. No retries permitted until 2026-10-31 00:55:52.102977526 +0000 UTC m=+3.264697166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5pdzc" (UniqueName: "kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc") pod "network-check-target-qhvwh" (UID: "74673206-8b68-4414-8ac0-a63577cea44a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:55:51.609048 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.605585    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vwtr\" (UniqueName: \"kubernetes.io/projected/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-kube-api-access-2vwtr\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:55:51.609048 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.605871    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfxdb\" (UniqueName: \"kubernetes.io/projected/68af8611-a641-4663-b13b-8f015107e82f-kube-api-access-kfxdb\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.609048 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.606864    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr7vl\" (UniqueName: \"kubernetes.io/projected/ac61d127-75ba-4f73-bcc6-7d69bf320a55-kube-api-access-vr7vl\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:55:51.609048 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.607402    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9mcc\" (UniqueName: \"kubernetes.io/projected/9bd7a453-d0b1-4db7-8dfa-247a72467481-kube-api-access-q9mcc\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.609894 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.609872    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-542pt\" (UniqueName: \"kubernetes.io/projected/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-kube-api-access-542pt\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:55:51.620352 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.620336    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7rmmh"
Oct 31 00:55:51.627947 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.627930    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:55:51.630869 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:51.630840    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12564994_4d47_4401_bc99_a4755b7ed33f.slice/crio-c2bba292241a8498175712a6eda34cd0491c8ef41e87e0a45f7425d8fdce9ff1 WatchSource:0}: Error finding container c2bba292241a8498175712a6eda34cd0491c8ef41e87e0a45f7425d8fdce9ff1: Status 404 returned error can't find the container with id c2bba292241a8498175712a6eda34cd0491c8ef41e87e0a45f7425d8fdce9ff1
Oct 31 00:55:51.635160 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.635137    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:55:51.645222 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.645200    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:55:51.646470 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:51.646447    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4311d66_bb16_4a76_a021_527f4c1395f4.slice/crio-e05e91dc951626d118ea0c64ef66f49a8b3a1e997129a15262652912c4d78815 WatchSource:0}: Error finding container e05e91dc951626d118ea0c64ef66f49a8b3a1e997129a15262652912c4d78815: Status 404 returned error can't find the container with id e05e91dc951626d118ea0c64ef66f49a8b3a1e997129a15262652912c4d78815
Oct 31 00:55:51.655964 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:51.655938    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c82039_550c_4fe8_96c7_dfd8990440e4.slice/crio-8a2a1f63e3e81d3ccdbdd2c6456fc992387964f779731a8c8285a71d41e76f9e WatchSource:0}: Error finding container 8a2a1f63e3e81d3ccdbdd2c6456fc992387964f779731a8c8285a71d41e76f9e: Status 404 returned error can't find the container with id 8a2a1f63e3e81d3ccdbdd2c6456fc992387964f779731a8c8285a71d41e76f9e
Oct 31 00:55:51.658206 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.658188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:55:51.665661 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.665643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:55:51.668655 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:51.668635    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac61d127_75ba_4f73_bcc6_7d69bf320a55.slice/crio-af00f6f9ab3bc53a0cbf6eb4a44ea51d728c9ffddc86ddfbc6817d7c46124b70 WatchSource:0}: Error finding container af00f6f9ab3bc53a0cbf6eb4a44ea51d728c9ffddc86ddfbc6817d7c46124b70: Status 404 returned error can't find the container with id af00f6f9ab3bc53a0cbf6eb4a44ea51d728c9ffddc86ddfbc6817d7c46124b70
Oct 31 00:55:51.671276 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.671247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:55:51.673776 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.673758    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:55:51.674383 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:51.674365    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21550fa_496b_4806_abd8_7f0441da9256.slice/crio-f2712ec09b95f839afc49b8f58afa9ff40d6674dac5109fb129f5c4946003e54 WatchSource:0}: Error finding container f2712ec09b95f839afc49b8f58afa9ff40d6674dac5109fb129f5c4946003e54: Status 404 returned error can't find the container with id f2712ec09b95f839afc49b8f58afa9ff40d6674dac5109fb129f5c4946003e54
Oct 31 00:55:51.682140 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:51.682121    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd7a453_d0b1_4db7_8dfa_247a72467481.slice/crio-e81be81de52ad909a8121a209c468ae0d5165819cf423fd3e1147acf31759647 WatchSource:0}: Error finding container e81be81de52ad909a8121a209c468ae0d5165819cf423fd3e1147acf31759647: Status 404 returned error can't find the container with id e81be81de52ad909a8121a209c468ae0d5165819cf423fd3e1147acf31759647
Oct 31 00:55:51.683998 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.683976    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:55:51.684988 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:51.684968    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae98732_67f4_420b_9e5b_e4e311ac9f9c.slice/crio-dfbbc42e46c5051dc778989248e094d6ffeb4f61582bc2f5d423a50012d9acc4 WatchSource:0}: Error finding container dfbbc42e46c5051dc778989248e094d6ffeb4f61582bc2f5d423a50012d9acc4: Status 404 returned error can't find the container with id dfbbc42e46c5051dc778989248e094d6ffeb4f61582bc2f5d423a50012d9acc4
Oct 31 00:55:51.693625 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:55:51.693575    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68af8611_a641_4663_b13b_8f015107e82f.slice/crio-490615ce9201733ff2579561f1eca8528fda8376df97bab8681e5b92a8143865 WatchSource:0}: Error finding container 490615ce9201733ff2579561f1eca8528fda8376df97bab8681e5b92a8143865: Status 404 returned error can't find the container with id 490615ce9201733ff2579561f1eca8528fda8376df97bab8681e5b92a8143865
Oct 31 00:55:51.948242 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:51.948138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:51.948428 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:51.948334    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:55:51.948428 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:51.948401    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret podName:e0cade88-4da4-467c-9656-cce6afe1dd62 nodeName:}" failed. No retries permitted until 2026-10-31 00:55:52.948383091 +0000 UTC m=+4.110102738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret") pod "global-pull-secret-syncer-jdmsp" (UID: "e0cade88-4da4-467c-9656-cce6afe1dd62") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:55:52.048841 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:52.048802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:55:52.049023 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:52.048934    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:55:52.049023 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:52.048985    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs podName:9d67aaf2-9211-4f6c-84d8-99aae85c28eb nodeName:}" failed. No retries permitted until 2026-10-31 00:55:53.048972179 +0000 UTC m=+4.210691835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs") pod "network-metrics-daemon-rlr9b" (UID: "9d67aaf2-9211-4f6c-84d8-99aae85c28eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:55:52.149814 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:52.149776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdzc\" (UniqueName: \"kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc\") pod \"network-check-target-qhvwh\" (UID: \"74673206-8b68-4414-8ac0-a63577cea44a\") " pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:55:52.150022 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:52.150001    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:55:52.150022 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:52.150022    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:55:52.150144 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:52.150033    2577 projected.go:196] Error preparing data for projected volume kube-api-access-5pdzc for pod openshift-network-diagnostics/network-check-target-qhvwh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:55:52.150144 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:52.150081    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc podName:74673206-8b68-4414-8ac0-a63577cea44a nodeName:}" failed. No retries permitted until 2026-10-31 00:55:53.150065774 +0000 UTC m=+4.311785432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5pdzc" (UniqueName: "kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc") pod "network-check-target-qhvwh" (UID: "74673206-8b68-4414-8ac0-a63577cea44a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:55:52.309444 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:52.309360    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r" event={"ID":"68af8611-a641-4663-b13b-8f015107e82f","Type":"ContainerStarted","Data":"490615ce9201733ff2579561f1eca8528fda8376df97bab8681e5b92a8143865"}
Oct 31 00:55:52.311108 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:52.311077    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n2x8d" event={"ID":"9bd7a453-d0b1-4db7-8dfa-247a72467481","Type":"ContainerStarted","Data":"e81be81de52ad909a8121a209c468ae0d5165819cf423fd3e1147acf31759647"}
Oct 31 00:55:52.312747 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:52.312717    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2k97" event={"ID":"f21550fa-496b-4806-abd8-7f0441da9256","Type":"ContainerStarted","Data":"f2712ec09b95f839afc49b8f58afa9ff40d6674dac5109fb129f5c4946003e54"}
Oct 31 00:55:52.314671 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:52.314632    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-djz6l" event={"ID":"ac61d127-75ba-4f73-bcc6-7d69bf320a55","Type":"ContainerStarted","Data":"af00f6f9ab3bc53a0cbf6eb4a44ea51d728c9ffddc86ddfbc6817d7c46124b70"}
Oct 31 00:55:52.316481 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:52.316451    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jww65" event={"ID":"7b893def-2fc6-4350-a01f-6ee4969c48bd","Type":"ContainerStarted","Data":"ff6b779da7560ae07af4993b0a32ba95bb04a57de51482b122b2a7f82f4c173c"}
Oct 31 00:55:52.318000 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:52.317965    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gsv6r" event={"ID":"7ae98732-67f4-420b-9e5b-e4e311ac9f9c","Type":"ContainerStarted","Data":"dfbbc42e46c5051dc778989248e094d6ffeb4f61582bc2f5d423a50012d9acc4"}
Oct 31 00:55:52.319367 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:52.319343    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj" event={"ID":"27c82039-550c-4fe8-96c7-dfd8990440e4","Type":"ContainerStarted","Data":"8a2a1f63e3e81d3ccdbdd2c6456fc992387964f779731a8c8285a71d41e76f9e"}
Oct 31 00:55:52.321164 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:52.321133    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lthm8" event={"ID":"e4311d66-bb16-4a76-a021-527f4c1395f4","Type":"ContainerStarted","Data":"e05e91dc951626d118ea0c64ef66f49a8b3a1e997129a15262652912c4d78815"}
Oct 31 00:55:52.322466 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:52.322442    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7rmmh" event={"ID":"12564994-4d47-4401-bc99-a4755b7ed33f","Type":"ContainerStarted","Data":"c2bba292241a8498175712a6eda34cd0491c8ef41e87e0a45f7425d8fdce9ff1"}
Oct 31 00:55:52.422875 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:52.422818    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:50:52 +0000 UTC" deadline="2028-06-02 14:06:11.961559907 +0000 UTC"
Oct 31 00:55:52.422875 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:52.422858    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13933h10m19.53870759s"
Oct 31 00:55:52.957112 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:52.957070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:52.957614 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:52.957590    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:55:52.957718 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:52.957697    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret podName:e0cade88-4da4-467c-9656-cce6afe1dd62 nodeName:}" failed. No retries permitted until 2026-10-31 00:55:54.95766371 +0000 UTC m=+6.119383355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret") pod "global-pull-secret-syncer-jdmsp" (UID: "e0cade88-4da4-467c-9656-cce6afe1dd62") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:55:53.058129 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:53.057915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:55:53.058129 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:53.058030    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:55:53.058129 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:53.058079    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs podName:9d67aaf2-9211-4f6c-84d8-99aae85c28eb nodeName:}" failed. No retries permitted until 2026-10-31 00:55:55.05806555 +0000 UTC m=+6.219785205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs") pod "network-metrics-daemon-rlr9b" (UID: "9d67aaf2-9211-4f6c-84d8-99aae85c28eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:55:53.158804 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:53.158766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdzc\" (UniqueName: \"kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc\") pod \"network-check-target-qhvwh\" (UID: \"74673206-8b68-4414-8ac0-a63577cea44a\") " pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:55:53.159070 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:53.159050    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:55:53.159144 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:53.159079    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:55:53.159144 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:53.159093    2577 projected.go:196] Error preparing data for projected volume kube-api-access-5pdzc for pod openshift-network-diagnostics/network-check-target-qhvwh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:55:53.159260 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:53.159154    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc podName:74673206-8b68-4414-8ac0-a63577cea44a nodeName:}" failed. No retries permitted until 2026-10-31 00:55:55.159133671 +0000 UTC m=+6.320853315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5pdzc" (UniqueName: "kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc") pod "network-check-target-qhvwh" (UID: "74673206-8b68-4414-8ac0-a63577cea44a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:55:53.304575 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:53.303302    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:55:53.304575 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:53.303441    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:55:53.304575 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:53.303541    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:53.304575 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:53.303630    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:55:53.304575 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:53.303686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:55:53.304575 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:53.303740    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:55:53.423407 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:53.423338    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:50:52 +0000 UTC" deadline="2028-06-21 19:28:16.612175376 +0000 UTC"
Oct 31 00:55:53.423407 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:53.423403    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14394h32m23.188776794s"
Oct 31 00:55:54.340432 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:54.340081    2577 generic.go:356] "Generic (PLEG): container finished" podID="52cc4e07643ef7c5669e5939f9f4241e" containerID="6e8ac3262e8d2f0bee1359580c2a7129de6fdd9d7d1a774a4c5299400700fdc9" exitCode=0
Oct 31 00:55:54.340432 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:54.340376    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal" event={"ID":"52cc4e07643ef7c5669e5939f9f4241e","Type":"ContainerDied","Data":"6e8ac3262e8d2f0bee1359580c2a7129de6fdd9d7d1a774a4c5299400700fdc9"}
Oct 31 00:55:54.977957 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:54.977591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:54.977957 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:54.977801    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:55:54.977957 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:54.977869    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret podName:e0cade88-4da4-467c-9656-cce6afe1dd62 nodeName:}" failed. No retries permitted until 2026-10-31 00:55:58.977849752 +0000 UTC m=+10.139569398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret") pod "global-pull-secret-syncer-jdmsp" (UID: "e0cade88-4da4-467c-9656-cce6afe1dd62") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:55:55.078834 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:55.078575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:55:55.078834 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:55.078732    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:55:55.078834 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:55.078798    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs podName:9d67aaf2-9211-4f6c-84d8-99aae85c28eb nodeName:}" failed. No retries permitted until 2026-10-31 00:55:59.078777202 +0000 UTC m=+10.240496850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs") pod "network-metrics-daemon-rlr9b" (UID: "9d67aaf2-9211-4f6c-84d8-99aae85c28eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:55:55.180268 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:55.179912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdzc\" (UniqueName: \"kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc\") pod \"network-check-target-qhvwh\" (UID: \"74673206-8b68-4414-8ac0-a63577cea44a\") " pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:55:55.180268 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:55.180100    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:55:55.180268 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:55.180121    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:55:55.180268 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:55.180134    2577 projected.go:196] Error preparing data for projected volume kube-api-access-5pdzc for pod openshift-network-diagnostics/network-check-target-qhvwh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:55:55.180268 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:55.180210    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc podName:74673206-8b68-4414-8ac0-a63577cea44a nodeName:}" failed. No retries permitted until 2026-10-31 00:55:59.180188927 +0000 UTC m=+10.341908575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5pdzc" (UniqueName: "kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc") pod "network-check-target-qhvwh" (UID: "74673206-8b68-4414-8ac0-a63577cea44a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:55:55.302758 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:55.302727    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:55.302947 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:55.302846    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:55:55.302947 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:55.302899    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:55:55.303071 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:55.302953    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:55:55.303071 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:55.302991    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:55:55.303071 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:55.303035    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:55:55.350032 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:55.349992    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2k97" event={"ID":"f21550fa-496b-4806-abd8-7f0441da9256","Type":"ContainerStarted","Data":"004cf2532b77c7fdb0155215d2b84048f53f7fadd1a3199ddf6eabf122137756"}
Oct 31 00:55:55.363215 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:55.363176    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal" event={"ID":"7fc072a2ae8e85ec3464a94d2bac4401","Type":"ContainerStarted","Data":"4e6e6b0b92803e591a81bc405bb426c636208ffce7fedf75efa0220644e87eeb"}
Oct 31 00:55:55.379909 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:55.379875    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal" event={"ID":"52cc4e07643ef7c5669e5939f9f4241e","Type":"ContainerStarted","Data":"09dd2b8b41fc4a0154584d33f7a3d43914179db9d812bb3591836b450042ca28"}
Oct 31 00:55:55.849231 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:55.848795    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal" podStartSLOduration=4.848770659 podStartE2EDuration="4.848770659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:55.725413572 +0000 UTC m=+6.887133239" watchObservedRunningTime="2026-10-31 00:55:55.848770659 +0000 UTC m=+7.010490315"
Oct 31 00:55:56.387742 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:56.386636    2577 generic.go:356] "Generic (PLEG): container finished" podID="f21550fa-496b-4806-abd8-7f0441da9256" containerID="004cf2532b77c7fdb0155215d2b84048f53f7fadd1a3199ddf6eabf122137756" exitCode=0
Oct 31 00:55:56.387742 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:56.386733    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2k97" event={"ID":"f21550fa-496b-4806-abd8-7f0441da9256","Type":"ContainerDied","Data":"004cf2532b77c7fdb0155215d2b84048f53f7fadd1a3199ddf6eabf122137756"}
Oct 31 00:55:56.393956 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:56.393498    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jww65" event={"ID":"7b893def-2fc6-4350-a01f-6ee4969c48bd","Type":"ContainerStarted","Data":"a6228e516163ec834ba03c7c0023fe06705f7825cbcdf21841bdf4b67d5c0201"}
Oct 31 00:55:56.621176 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:56.621108    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal" podStartSLOduration=5.621090597 podStartE2EDuration="5.621090597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:55.857872827 +0000 UTC m=+7.019592504" watchObservedRunningTime="2026-10-31 00:55:56.621090597 +0000 UTC m=+7.782810638"
Oct 31 00:55:57.302835 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:57.302802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:55:57.303033 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:57.302959    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:55:57.303104 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:57.303043    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:55:57.303171 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:57.303117    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:55:57.303244 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:57.303199    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:57.303381 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:57.303298    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:55:58.402398 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:58.402336    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj" event={"ID":"27c82039-550c-4fe8-96c7-dfd8990440e4","Type":"ContainerStarted","Data":"cba33337d7d66c8d578a39273ca84a1f7b53a7eb8df49bf48dab74cbaa453c90"}
Oct 31 00:55:58.407243 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:58.407213    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lthm8" event={"ID":"e4311d66-bb16-4a76-a021-527f4c1395f4","Type":"ContainerStarted","Data":"04f0bedd3893d1f388d06de87840e8eb47bf849732db4381c8116c7f10954d7b"}
Oct 31 00:55:58.532175 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:58.532112    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jww65" podStartSLOduration=4.045512549 podStartE2EDuration="8.5320909s" totalImagesPullingTime="4.486578351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:56.745412034 +0000 UTC m=+7.907131696" watchObservedRunningTime="2026-10-31 00:55:58.5320909 +0000 UTC m=+9.693810568"
Oct 31 00:55:59.011123 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:59.011075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:59.011345 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:59.011273    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:55:59.011345 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:59.011340    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret podName:e0cade88-4da4-467c-9656-cce6afe1dd62 nodeName:}" failed. No retries permitted until 2026-10-31 00:56:07.011320987 +0000 UTC m=+18.173040629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret") pod "global-pull-secret-syncer-jdmsp" (UID: "e0cade88-4da4-467c-9656-cce6afe1dd62") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:55:59.111697 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:59.111653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:55:59.111881 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:59.111834    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:55:59.111956 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:59.111931    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs podName:9d67aaf2-9211-4f6c-84d8-99aae85c28eb nodeName:}" failed. No retries permitted until 2026-10-31 00:56:07.111908985 +0000 UTC m=+18.273628628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs") pod "network-metrics-daemon-rlr9b" (UID: "9d67aaf2-9211-4f6c-84d8-99aae85c28eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:55:59.212503 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:59.212458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdzc\" (UniqueName: \"kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc\") pod \"network-check-target-qhvwh\" (UID: \"74673206-8b68-4414-8ac0-a63577cea44a\") " pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:55:59.212685 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:59.212657    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:55:59.212685 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:59.212682    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:55:59.212787 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:59.212697    2577 projected.go:196] Error preparing data for projected volume kube-api-access-5pdzc for pod openshift-network-diagnostics/network-check-target-qhvwh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:55:59.212787 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:59.212769    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc podName:74673206-8b68-4414-8ac0-a63577cea44a nodeName:}" failed. No retries permitted until 2026-10-31 00:56:07.212744833 +0000 UTC m=+18.374464483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5pdzc" (UniqueName: "kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc") pod "network-check-target-qhvwh" (UID: "74673206-8b68-4414-8ac0-a63577cea44a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:55:59.304887 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:59.304775    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:55:59.306971 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:59.305054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:55:59.307154 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:59.305085    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:55:59.307710 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:59.307657    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:55:59.307886 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:59.307743    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:55:59.308580 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:55:59.307895    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:55:59.684473 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:59.683734    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:55:59.684954 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:59.684621    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:55:59.828788 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:55:59.828282    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lthm8" podStartSLOduration=3.768610097 podStartE2EDuration="9.828242433s" totalImagesPullingTime="6.059632336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:58.532459356 +0000 UTC m=+9.694179021" watchObservedRunningTime="2026-10-31 00:55:59.828242433 +0000 UTC m=+10.989962099"
Oct 31 00:56:00.555503 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:00.555475    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:56:00.812855 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:00.812699    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:56:00.813301 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:00.813263    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:56:01.300311 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:01.300274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:01.300432 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:01.300381    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:56:01.300487 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:01.300438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:01.300533 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:01.300485    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:56:01.300605 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:01.300594    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:01.300680 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:01.300667    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:56:01.302140 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:01.302046    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:56:00.55550173Z","UUID":"cadaddc6-18ab-4b03-9957-7b05f1429666","Handler":null,"Name":"","Endpoint":""}
Oct 31 00:56:01.303902 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:01.303687    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 00:56:01.303902 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:01.303730    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:56:01.416390 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:01.416358    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj" event={"ID":"27c82039-550c-4fe8-96c7-dfd8990440e4","Type":"ContainerStarted","Data":"d4ed9ee71a162ee6e489ebc84650f011ab7815ecb8ba29083b90114213c7868e"}
Oct 31 00:56:01.419222 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:01.418899    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7rmmh" event={"ID":"12564994-4d47-4401-bc99-a4755b7ed33f","Type":"ContainerStarted","Data":"b2100cdbf5b041bf7b7021d43ada162d0ab33b081187bdf0c1947b9410e3d4a0"}
Oct 31 00:56:01.421913 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:01.421576    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2k97" event={"ID":"f21550fa-496b-4806-abd8-7f0441da9256","Type":"ContainerStarted","Data":"f472e72a2dad76861bbe191902e76df78f2a5ad861c4f72067d20ae87b24112b"}
Oct 31 00:56:01.423963 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:01.423940    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gsv6r" event={"ID":"7ae98732-67f4-420b-9e5b-e4e311ac9f9c","Type":"ContainerStarted","Data":"5d0f6d13e2b666afd2287fe81a8d9aa1ba9a188f74ec605baaafb28025fec01b"}
Oct 31 00:56:01.548304 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:01.546911    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7rmmh" podStartSLOduration=2.725539507 podStartE2EDuration="11.546889681s" totalImagesPullingTime="8.821350174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:01.544670828 +0000 UTC m=+12.706390491" watchObservedRunningTime="2026-10-31 00:56:01.546889681 +0000 UTC m=+12.708609343"
Oct 31 00:56:02.428422 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:02.427852    2577 generic.go:356] "Generic (PLEG): container finished" podID="f21550fa-496b-4806-abd8-7f0441da9256" containerID="f472e72a2dad76861bbe191902e76df78f2a5ad861c4f72067d20ae87b24112b" exitCode=0
Oct 31 00:56:02.428422 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:02.427901    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2k97" event={"ID":"f21550fa-496b-4806-abd8-7f0441da9256","Type":"ContainerDied","Data":"f472e72a2dad76861bbe191902e76df78f2a5ad861c4f72067d20ae87b24112b"}
Oct 31 00:56:02.430787 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:02.430763    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-djz6l" event={"ID":"ac61d127-75ba-4f73-bcc6-7d69bf320a55","Type":"ContainerStarted","Data":"1f10bad8581f59cb5b945976ef834b29e32660c75c0deed0cb0533324ae868ee"}
Oct 31 00:56:02.433644 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:02.433617    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj" event={"ID":"27c82039-550c-4fe8-96c7-dfd8990440e4","Type":"ContainerStarted","Data":"cb8a6b1ba085ffeae6db227fe10d833873477f1f99619443fdf9a7f0667d825a"}
Oct 31 00:56:02.546768 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:02.546719    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gsv6r" podStartSLOduration=3.573441522 podStartE2EDuration="12.546701659s" totalImagesPullingTime="8.973260137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:01.786563525 +0000 UTC m=+12.948283176" watchObservedRunningTime="2026-10-31 00:56:02.546701659 +0000 UTC m=+13.708421321"
Oct 31 00:56:02.781277 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:02.781188    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj" podStartSLOduration=2.584255341 podStartE2EDuration="12.781160972s" totalImagesPullingTime="10.196905631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:02.665829661 +0000 UTC m=+13.827549323" watchObservedRunningTime="2026-10-31 00:56:02.781160972 +0000 UTC m=+13.942880633"
Oct 31 00:56:02.781476 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:02.781412    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-djz6l" podStartSLOduration=3.688947666 podStartE2EDuration="12.781406934s" totalImagesPullingTime="9.092459268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:02.781047046 +0000 UTC m=+13.942766708" watchObservedRunningTime="2026-10-31 00:56:02.781406934 +0000 UTC m=+13.943126594"
Oct 31 00:56:03.302158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:03.302131    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:03.302320 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:03.302290    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:56:03.302375 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:03.302367    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:03.302620 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:03.302434    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:56:03.302620 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:03.302487    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:03.302620 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:03.302554    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:56:03.438573 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:03.438542    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n2x8d" event={"ID":"9bd7a453-d0b1-4db7-8dfa-247a72467481","Type":"ContainerStarted","Data":"598f3b56e1f5ec93ef6c31d31ed89ac24ddba71c2a2a459beaaae5a09a266b18"}
Oct 31 00:56:04.442019 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:04.441975    2577 generic.go:356] "Generic (PLEG): container finished" podID="f21550fa-496b-4806-abd8-7f0441da9256" containerID="e368fed69a55bc1943e2981ee75c1d43ad2757da847dd3112c779423c6a371cc" exitCode=0
Oct 31 00:56:04.442542 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:04.442061    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2k97" event={"ID":"f21550fa-496b-4806-abd8-7f0441da9256","Type":"ContainerDied","Data":"e368fed69a55bc1943e2981ee75c1d43ad2757da847dd3112c779423c6a371cc"}
Oct 31 00:56:04.561131 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:04.561084    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n2x8d" podStartSLOduration=3.006544578 podStartE2EDuration="14.561070728s" totalImagesPullingTime="11.55452615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:03.558322854 +0000 UTC m=+14.720042533" watchObservedRunningTime="2026-10-31 00:56:04.561070728 +0000 UTC m=+15.722790389"
Oct 31 00:56:05.299885 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:05.299851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:05.300070 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:05.299895    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:05.300070 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:05.300029    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:56:05.300230 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:05.300104    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:56:05.300230 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:05.300198    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:05.308961 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:05.300280    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:56:06.449115 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:06.448856    2577 generic.go:356] "Generic (PLEG): container finished" podID="f21550fa-496b-4806-abd8-7f0441da9256" containerID="4f094334431c240d11fe029d8b19cdaf3f2c6061c986816323fa100072fe0bb1" exitCode=0
Oct 31 00:56:06.449115 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:06.448954    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2k97" event={"ID":"f21550fa-496b-4806-abd8-7f0441da9256","Type":"ContainerDied","Data":"4f094334431c240d11fe029d8b19cdaf3f2c6061c986816323fa100072fe0bb1"}
Oct 31 00:56:07.090991 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:07.090959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:07.091208 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:07.091124    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:56:07.091250 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:07.091210    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret podName:e0cade88-4da4-467c-9656-cce6afe1dd62 nodeName:}" failed. No retries permitted until 2026-10-31 00:56:23.091194014 +0000 UTC m=+34.252913665 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret") pod "global-pull-secret-syncer-jdmsp" (UID: "e0cade88-4da4-467c-9656-cce6afe1dd62") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:56:07.191961 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:07.191928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:07.192098 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:07.192049    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:56:07.192151 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:07.192106    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs podName:9d67aaf2-9211-4f6c-84d8-99aae85c28eb nodeName:}" failed. No retries permitted until 2026-10-31 00:56:23.192091443 +0000 UTC m=+34.353811086 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs") pod "network-metrics-daemon-rlr9b" (UID: "9d67aaf2-9211-4f6c-84d8-99aae85c28eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:56:07.292389 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:07.292357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdzc\" (UniqueName: \"kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc\") pod \"network-check-target-qhvwh\" (UID: \"74673206-8b68-4414-8ac0-a63577cea44a\") " pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:07.292539 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:07.292511    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:56:07.292539 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:07.292531    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:56:07.292539 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:07.292540    2577 projected.go:196] Error preparing data for projected volume kube-api-access-5pdzc for pod openshift-network-diagnostics/network-check-target-qhvwh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:56:07.292624 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:07.292587    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc podName:74673206-8b68-4414-8ac0-a63577cea44a nodeName:}" failed. No retries permitted until 2026-10-31 00:56:23.292573675 +0000 UTC m=+34.454293313 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5pdzc" (UniqueName: "kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc") pod "network-check-target-qhvwh" (UID: "74673206-8b68-4414-8ac0-a63577cea44a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:56:07.300595 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:07.300569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:07.300595 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:07.300593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:07.300757 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:07.300621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:07.300757 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:07.300706    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:56:07.300757 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:07.300747    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:56:07.300907 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:07.300824    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:56:08.455742 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:08.455572    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r" event={"ID":"68af8611-a641-4663-b13b-8f015107e82f","Type":"ContainerStarted","Data":"82134f4d9492446f11cd056733704ba6ebcccaf9155cdbe5cbbc131e33966bb6"}
Oct 31 00:56:08.456132 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:08.455796    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r" event={"ID":"68af8611-a641-4663-b13b-8f015107e82f","Type":"ContainerStarted","Data":"7e3c428496c1e530902d90feab84f4722b91cea601f6dd5a0a70bc44902d2ca2"}
Oct 31 00:56:08.456132 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:08.455850    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r" event={"ID":"68af8611-a641-4663-b13b-8f015107e82f","Type":"ContainerStarted","Data":"957f1590b26bc1fae659456a59173bc9f23a16f0b5e545c1681fe67786dfcc28"}
Oct 31 00:56:08.456132 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:08.455905    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r" event={"ID":"68af8611-a641-4663-b13b-8f015107e82f","Type":"ContainerStarted","Data":"12887fe3483d508f2cacc690ab35c2d8e791a181e249a5e6ca23ee4febbce990"}
Oct 31 00:56:08.456132 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:08.455952    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r" event={"ID":"68af8611-a641-4663-b13b-8f015107e82f","Type":"ContainerStarted","Data":"780ce14e38081b47d489aa3d7ba58560382a551f04f62325f7c25afbd80292c1"}
Oct 31 00:56:08.456132 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:08.455999    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r" event={"ID":"68af8611-a641-4663-b13b-8f015107e82f","Type":"ContainerStarted","Data":"d919df306a18eab7a160adf0dfeec900bca6a5e4a0305c878221d4e22b63c89c"}
Oct 31 00:56:09.300540 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:09.300516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:09.300686 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:09.300588    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:56:09.300686 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:09.300606    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:09.300746 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:09.300731    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:56:09.300817 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:09.300804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:09.300900 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:09.300884    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:56:09.460053 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:09.460021    2577 generic.go:356] "Generic (PLEG): container finished" podID="f21550fa-496b-4806-abd8-7f0441da9256" containerID="f6cfb2aba65d5ff5f3e86cb1bfb489fbafb59488e4ae5a2fde5b45218d975716" exitCode=0
Oct 31 00:56:09.460519 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:09.460066    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2k97" event={"ID":"f21550fa-496b-4806-abd8-7f0441da9256","Type":"ContainerDied","Data":"f6cfb2aba65d5ff5f3e86cb1bfb489fbafb59488e4ae5a2fde5b45218d975716"}
Oct 31 00:56:10.465586 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:10.465555    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r" event={"ID":"68af8611-a641-4663-b13b-8f015107e82f","Type":"ContainerStarted","Data":"5f7ea302532c6b0650c25d449e367be0fbd7abbc62adf3c85e79906f45c49584"}
Oct 31 00:56:10.468363 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:10.468337    2577 generic.go:356] "Generic (PLEG): container finished" podID="f21550fa-496b-4806-abd8-7f0441da9256" containerID="724b362628522850930523214c51dcab8af71b86684609170c6c5bf481af5cd9" exitCode=0
Oct 31 00:56:10.468458 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:10.468375    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2k97" event={"ID":"f21550fa-496b-4806-abd8-7f0441da9256","Type":"ContainerDied","Data":"724b362628522850930523214c51dcab8af71b86684609170c6c5bf481af5cd9"}
Oct 31 00:56:11.299783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:11.299749    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:11.299783 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:11.299778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:11.300023 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:11.299879    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:56:11.300023 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:11.299914    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:11.300023 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:11.299989    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:56:11.300149 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:11.300041    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:56:11.472988 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:11.472956    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2k97" event={"ID":"f21550fa-496b-4806-abd8-7f0441da9256","Type":"ContainerStarted","Data":"df5c987ffc90f8983f977e213e1e33ade0366efd9846a22a217cfaf333f875e1"}
Oct 31 00:56:12.478532 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:12.478491    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r" event={"ID":"68af8611-a641-4663-b13b-8f015107e82f","Type":"ContainerStarted","Data":"e87511ad84209dcdba0201937e3212b442559431cb6f6c57f8d150c86f11240e"}
Oct 31 00:56:12.774242 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:12.774099    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g2k97" podStartSLOduration=10.277615915 podStartE2EDuration="22.774083929s" totalImagesPullingTime="12.496468014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:11.590984114 +0000 UTC m=+22.752703774" watchObservedRunningTime="2026-10-31 00:56:12.774083929 +0000 UTC m=+23.935803590"
Oct 31 00:56:12.774382 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:12.774322    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r" podStartSLOduration=6.992954321 podStartE2EDuration="22.774316209s" totalImagesPullingTime="15.781361888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:12.773802628 +0000 UTC m=+23.935522290" watchObservedRunningTime="2026-10-31 00:56:12.774316209 +0000 UTC m=+23.936035870"
Oct 31 00:56:13.299802 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:13.299771    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:13.299802 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:13.299793    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:13.300016 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:13.299892    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:13.300016 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:13.299901    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:56:13.300016 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:13.299956    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:56:13.300113 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:13.300032    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:56:13.481059 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:13.481021    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:13.481516 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:13.481101    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:13.481516 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:13.481143    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:13.497014 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:13.496870    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:13.497144 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:13.497121    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:14.753755 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:14.753651    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jdmsp"]
Oct 31 00:56:14.754129 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:14.753801    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:14.754129 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:14.753901    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:56:14.756881 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:14.756858    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qhvwh"]
Oct 31 00:56:14.756994 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:14.756960    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:14.757073 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:14.757053    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:56:14.757566 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:14.757544    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rlr9b"]
Oct 31 00:56:14.757664 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:14.757638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:14.757733 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:14.757715    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:56:15.103075 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:15.103002    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:15.407841 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:15.407754    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:16.299685 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:16.299603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:16.299685 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:16.299652    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:16.300237 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:16.299727    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:16.300237 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:16.299747    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:56:16.300237 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:16.299791    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:56:16.300237 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:16.299856    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:56:16.451030 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:16.451004    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:18.300339 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:18.300310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:18.300726 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:18.300395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:18.300726 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:18.300421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:18.300726 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:18.300485    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:56:18.300726 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:18.300540    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:56:18.300726 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:18.300651    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:56:20.005665 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.005626    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:20.189160 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.189132    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:20.300594 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.300566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:20.300769 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.300681    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:20.300769 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:20.300689    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlr9b" podUID="9d67aaf2-9211-4f6c-84d8-99aae85c28eb"
Oct 31 00:56:20.300769 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:20.300762    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvwh" podUID="74673206-8b68-4414-8ac0-a63577cea44a"
Oct 31 00:56:20.300929 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.300774    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:20.300929 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:20.300870    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jdmsp" podUID="e0cade88-4da4-467c-9656-cce6afe1dd62"
Oct 31 00:56:20.670159 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.670090    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeReady"
Oct 31 00:56:20.670316 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.670245    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 00:56:20.833660 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.833625    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dljpn"]
Oct 31 00:56:20.845885 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.845853    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5lczz"]
Oct 31 00:56:20.846064 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.845981    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dljpn"
Oct 31 00:56:20.850506 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.850483    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dljpn"]
Oct 31 00:56:20.850627 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.850516    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5lczz"]
Oct 31 00:56:20.850627 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.850585    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:20.898415 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.898383    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 00:56:20.898597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.898502    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sk5g6\""
Oct 31 00:56:20.898597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.898565    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 00:56:20.898597 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.898564    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 00:56:20.902087 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.902071    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 00:56:20.902158 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.902099    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 00:56:20.902438 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.902424    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rf2ln\""
Oct 31 00:56:20.985380 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.985311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-config-volume\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:20.985380 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.985367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e39c2f85-6b3b-4343-bb84-c531243a36e5-cert\") pod \"ingress-canary-dljpn\" (UID: \"e39c2f85-6b3b-4343-bb84-c531243a36e5\") " pod="openshift-ingress-canary/ingress-canary-dljpn"
Oct 31 00:56:20.985612 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.985455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:20.985612 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.985531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-tmp-dir\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:20.985688 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.985610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdcx7\" (UniqueName: \"kubernetes.io/projected/e39c2f85-6b3b-4343-bb84-c531243a36e5-kube-api-access-jdcx7\") pod \"ingress-canary-dljpn\" (UID: \"e39c2f85-6b3b-4343-bb84-c531243a36e5\") " pod="openshift-ingress-canary/ingress-canary-dljpn"
Oct 31 00:56:20.985688 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:20.985629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shw5k\" (UniqueName: \"kubernetes.io/projected/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-kube-api-access-shw5k\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:21.086301 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.086273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e39c2f85-6b3b-4343-bb84-c531243a36e5-cert\") pod \"ingress-canary-dljpn\" (UID: \"e39c2f85-6b3b-4343-bb84-c531243a36e5\") " pod="openshift-ingress-canary/ingress-canary-dljpn"
Oct 31 00:56:21.086798 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.086314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:21.086798 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.086332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-tmp-dir\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:21.086798 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.086350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdcx7\" (UniqueName: \"kubernetes.io/projected/e39c2f85-6b3b-4343-bb84-c531243a36e5-kube-api-access-jdcx7\") pod \"ingress-canary-dljpn\" (UID: \"e39c2f85-6b3b-4343-bb84-c531243a36e5\") " pod="openshift-ingress-canary/ingress-canary-dljpn"
Oct 31 00:56:21.086798 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.086367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shw5k\" (UniqueName: \"kubernetes.io/projected/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-kube-api-access-shw5k\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:21.086798 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:21.086452    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:56:21.086798 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.086495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-config-volume\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:21.086798 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:21.086523    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls podName:f90cbecc-43bb-41a8-b743-4cdcd12dce2f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:21.586503423 +0000 UTC m=+32.748223083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls") pod "dns-default-5lczz" (UID: "f90cbecc-43bb-41a8-b743-4cdcd12dce2f") : secret "dns-default-metrics-tls" not found
Oct 31 00:56:21.087086 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.086922    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-tmp-dir\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:21.087234 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.087169    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-config-volume\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:21.090567 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.090539    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e39c2f85-6b3b-4343-bb84-c531243a36e5-cert\") pod \"ingress-canary-dljpn\" (UID: \"e39c2f85-6b3b-4343-bb84-c531243a36e5\") " pod="openshift-ingress-canary/ingress-canary-dljpn"
Oct 31 00:56:21.151574 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.151539    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shw5k\" (UniqueName: \"kubernetes.io/projected/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-kube-api-access-shw5k\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:21.154254 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.154230    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdcx7\" (UniqueName: \"kubernetes.io/projected/e39c2f85-6b3b-4343-bb84-c531243a36e5-kube-api-access-jdcx7\") pod \"ingress-canary-dljpn\" (UID: \"e39c2f85-6b3b-4343-bb84-c531243a36e5\") " pod="openshift-ingress-canary/ingress-canary-dljpn"
Oct 31 00:56:21.160081 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.160057    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dljpn"
Oct 31 00:56:21.192698 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.192545    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:21.420150 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.420120    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:21.584617 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.584590    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dljpn"]
Oct 31 00:56:21.588102 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:56:21.588069    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39c2f85_6b3b_4343_bb84_c531243a36e5.slice/crio-e815def07a2cef6fc6b1bb2468d81241584a117d5ed3b335cd1611d5886f2903 WatchSource:0}: Error finding container e815def07a2cef6fc6b1bb2468d81241584a117d5ed3b335cd1611d5886f2903: Status 404 returned error can't find the container with id e815def07a2cef6fc6b1bb2468d81241584a117d5ed3b335cd1611d5886f2903
Oct 31 00:56:21.589218 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:21.589194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:21.589302 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:21.589280    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:56:21.589359 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:21.589349    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls podName:f90cbecc-43bb-41a8-b743-4cdcd12dce2f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:22.589331493 +0000 UTC m=+33.751051146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls") pod "dns-default-5lczz" (UID: "f90cbecc-43bb-41a8-b743-4cdcd12dce2f") : secret "dns-default-metrics-tls" not found
Oct 31 00:56:22.299630 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:22.299600    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:22.300111 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:22.299703    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:22.300111 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:22.299735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:22.352346 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:22.352312    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 00:56:22.352489 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:22.352399    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 00:56:22.402022 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:22.401994    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 00:56:22.402224 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:22.402066    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hnbxm\""
Oct 31 00:56:22.402224 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:22.402154    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vxv7v\""
Oct 31 00:56:22.402762 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:22.402746    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gjj6f\""
Oct 31 00:56:22.402891 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:22.402880    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 00:56:22.502122 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:22.502088    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dljpn" event={"ID":"e39c2f85-6b3b-4343-bb84-c531243a36e5","Type":"ContainerStarted","Data":"e815def07a2cef6fc6b1bb2468d81241584a117d5ed3b335cd1611d5886f2903"}
Oct 31 00:56:22.596633 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:22.596541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:22.596799 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:22.596781    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:56:22.596868 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:22.596856    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls podName:f90cbecc-43bb-41a8-b743-4cdcd12dce2f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:24.59683815 +0000 UTC m=+35.758557803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls") pod "dns-default-5lczz" (UID: "f90cbecc-43bb-41a8-b743-4cdcd12dce2f") : secret "dns-default-metrics-tls" not found
Oct 31 00:56:23.101423 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.101388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:23.103259 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.103230    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vqn7z"]
Oct 31 00:56:23.108819 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.108789    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:23.120704 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.120670    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.193773 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.193732    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vqn7z"]
Oct 31 00:56:23.201902 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.201874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:23.202069 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:23.202045    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:56:23.202137 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:23.202118    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs podName:9d67aaf2-9211-4f6c-84d8-99aae85c28eb nodeName:}" failed. No retries permitted until 2026-10-31 00:56:55.202099342 +0000 UTC m=+66.363818981 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs") pod "network-metrics-daemon-rlr9b" (UID: "9d67aaf2-9211-4f6c-84d8-99aae85c28eb") : secret "metrics-daemon-secret" not found
Oct 31 00:56:23.233600 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.233564    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 00:56:23.233815 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.233618    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 00:56:23.233815 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.233635    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 00:56:23.233815 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.233660    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 00:56:23.233815 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.233805    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m5b2l\""
Oct 31 00:56:23.237102 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.237074    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:23.292295 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.292254    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-dsx4h-8b9gr"]
Oct 31 00:56:23.302694 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.302663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-data-volume\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.303132 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.302707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47hxq\" (UniqueName: \"kubernetes.io/projected/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-api-access-47hxq\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.303132 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.302874    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.303132 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.302918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.303132 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.302957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-crio-socket\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.303132 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.302989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdzc\" (UniqueName: \"kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc\") pod \"network-check-target-qhvwh\" (UID: \"74673206-8b68-4414-8ac0-a63577cea44a\") " pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:23.312774 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.312752    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pdzc\" (UniqueName: \"kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc\") pod \"network-check-target-qhvwh\" (UID: \"74673206-8b68-4414-8ac0-a63577cea44a\") " pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:23.318376 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.318346    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:23.321808 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.321789    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:23.352371 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.352299    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dsx4h-8b9gr"]
Oct 31 00:56:23.373531 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.373487    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dj4kh\""
Oct 31 00:56:23.373714 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.373554    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 00:56:23.404223 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.404166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-data-volume\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.404385 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.404236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47hxq\" (UniqueName: \"kubernetes.io/projected/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-api-access-47hxq\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.404385 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.404305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.404385 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:23.404378    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:56:23.404515 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:23.404450    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm podName:7bf35a3c-66f9-4ccf-b340-50a072ffa61f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:23.904429177 +0000 UTC m=+35.066148828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vqn7z" (UID: "7bf35a3c-66f9-4ccf-b340-50a072ffa61f") : configmap "kube-rbac-proxy" not found
Oct 31 00:56:23.404581 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.404566    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-data-volume\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.404635 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.404567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.404635 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.404616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-crio-socket\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.404731 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:23.404675    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:56:23.404731 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:23.404713    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls podName:7bf35a3c-66f9-4ccf-b340-50a072ffa61f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:23.90470177 +0000 UTC m=+35.066421423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vqn7z" (UID: "7bf35a3c-66f9-4ccf-b340-50a072ffa61f") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:56:23.404838 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.404813    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-crio-socket\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.424000 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.423966    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 00:56:23.463036 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.463006    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47hxq\" (UniqueName: \"kubernetes.io/projected/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-api-access-47hxq\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.505092 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.505061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrm7b\" (UniqueName: \"kubernetes.io/projected/506d546e-6187-4c6a-967b-09273e1fa785-kube-api-access-nrm7b\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:23.505269 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.505114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-trusted-ca-bundle\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:23.505269 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.505139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-service-ca-bundle\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:23.505557 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.505293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/506d546e-6187-4c6a-967b-09273e1fa785-archives-path\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:23.553115 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.553083    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:23.556874 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.556848    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:23.605864 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.605796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/506d546e-6187-4c6a-967b-09273e1fa785-archives-path\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:23.605997 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.605867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrm7b\" (UniqueName: \"kubernetes.io/projected/506d546e-6187-4c6a-967b-09273e1fa785-kube-api-access-nrm7b\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:23.605997 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.605917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-trusted-ca-bundle\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:23.606404 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.606254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-service-ca-bundle\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:23.606404 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.606256    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/506d546e-6187-4c6a-967b-09273e1fa785-archives-path\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:23.606600 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.606579    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-service-ca-bundle\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:23.606857 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.606835    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-trusted-ca-bundle\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:23.617540 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.617510    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jdmsp"]
Oct 31 00:56:23.620861 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:56:23.620832    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cade88_4da4_467c_9656_cce6afe1dd62.slice/crio-6e45f6dff91a07a8b722c2deb45bb14c00b1e539af4914ad78a6d6c1b93bdb66 WatchSource:0}: Error finding container 6e45f6dff91a07a8b722c2deb45bb14c00b1e539af4914ad78a6d6c1b93bdb66: Status 404 returned error can't find the container with id 6e45f6dff91a07a8b722c2deb45bb14c00b1e539af4914ad78a6d6c1b93bdb66
Oct 31 00:56:23.664078 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.664047    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrm7b\" (UniqueName: \"kubernetes.io/projected/506d546e-6187-4c6a-967b-09273e1fa785-kube-api-access-nrm7b\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:23.749508 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.749465    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:23.909160 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.909075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.909160 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.909112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:23.909356 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:23.909241    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:56:23.909356 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:23.909276    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:56:23.909356 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:23.909311    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm podName:7bf35a3c-66f9-4ccf-b340-50a072ffa61f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:24.909293708 +0000 UTC m=+36.071013363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vqn7z" (UID: "7bf35a3c-66f9-4ccf-b340-50a072ffa61f") : configmap "kube-rbac-proxy" not found
Oct 31 00:56:23.909356 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:23.909326    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls podName:7bf35a3c-66f9-4ccf-b340-50a072ffa61f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:24.909320105 +0000 UTC m=+36.071039744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vqn7z" (UID: "7bf35a3c-66f9-4ccf-b340-50a072ffa61f") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:56:23.954205 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.954162    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:23.959354 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:23.959331    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qhvwh"]
Oct 31 00:56:23.962513 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:56:23.962486    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74673206_8b68_4414_8ac0_a63577cea44a.slice/crio-31d5f54f34e5d2a974970231e806aa8341c2b7b385a44854d947bf56215d519f WatchSource:0}: Error finding container 31d5f54f34e5d2a974970231e806aa8341c2b7b385a44854d947bf56215d519f: Status 404 returned error can't find the container with id 31d5f54f34e5d2a974970231e806aa8341c2b7b385a44854d947bf56215d519f
Oct 31 00:56:24.336920 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.336884    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dsx4h-8b9gr"]
Oct 31 00:56:24.340955 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:56:24.340928    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506d546e_6187_4c6a_967b_09273e1fa785.slice/crio-35e2c18924ca3bf73037fd4b2b57bcd54be0ecc893851fa01325cdef2cd9ffa2 WatchSource:0}: Error finding container 35e2c18924ca3bf73037fd4b2b57bcd54be0ecc893851fa01325cdef2cd9ffa2: Status 404 returned error can't find the container with id 35e2c18924ca3bf73037fd4b2b57bcd54be0ecc893851fa01325cdef2cd9ffa2
Oct 31 00:56:24.508889 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.508853    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dsx4h-8b9gr" event={"ID":"506d546e-6187-4c6a-967b-09273e1fa785","Type":"ContainerStarted","Data":"35e2c18924ca3bf73037fd4b2b57bcd54be0ecc893851fa01325cdef2cd9ffa2"}
Oct 31 00:56:24.510273 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.510242    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qhvwh" event={"ID":"74673206-8b68-4414-8ac0-a63577cea44a","Type":"ContainerStarted","Data":"31d5f54f34e5d2a974970231e806aa8341c2b7b385a44854d947bf56215d519f"}
Oct 31 00:56:24.511486 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.511460    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jdmsp" event={"ID":"e0cade88-4da4-467c-9656-cce6afe1dd62","Type":"ContainerStarted","Data":"6e45f6dff91a07a8b722c2deb45bb14c00b1e539af4914ad78a6d6c1b93bdb66"}
Oct 31 00:56:24.573333 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.573301    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9l9rr"]
Oct 31 00:56:24.584731 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.584697    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:24.616166 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.616131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:24.616343 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:24.616305    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:56:24.616418 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:24.616387    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls podName:f90cbecc-43bb-41a8-b743-4cdcd12dce2f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:28.616364262 +0000 UTC m=+39.778083923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls") pod "dns-default-5lczz" (UID: "f90cbecc-43bb-41a8-b743-4cdcd12dce2f") : secret "dns-default-metrics-tls" not found
Oct 31 00:56:24.634294 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.634261    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9l9rr"]
Oct 31 00:56:24.642608 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.642584    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 00:56:24.642763 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.642608    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 00:56:24.642763 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.642663    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 00:56:24.643292 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.643256    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 00:56:24.643426 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.643387    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nktwg\""
Oct 31 00:56:24.716680 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.716641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-signing-key\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:24.716864 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.716750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wd9l\" (UniqueName: \"kubernetes.io/projected/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-kube-api-access-4wd9l\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:24.716864 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.716780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-signing-cabundle\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:24.756054 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.756000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:24.817844 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.817807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-signing-key\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:24.818018 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.817902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wd9l\" (UniqueName: \"kubernetes.io/projected/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-kube-api-access-4wd9l\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:24.818018 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.817934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-signing-cabundle\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:24.818700 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.818675    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-signing-cabundle\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:24.821462 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.821438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-signing-key\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:24.876385 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.876308    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wd9l\" (UniqueName: \"kubernetes.io/projected/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-kube-api-access-4wd9l\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:24.908866 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.908823    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:24.919121 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.919084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:24.919321 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.919139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:24.919321 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:24.919253    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:56:24.919321 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:24.919285    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:56:24.919461 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:24.919339    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm podName:7bf35a3c-66f9-4ccf-b340-50a072ffa61f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:26.919317816 +0000 UTC m=+38.081037462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vqn7z" (UID: "7bf35a3c-66f9-4ccf-b340-50a072ffa61f") : configmap "kube-rbac-proxy" not found
Oct 31 00:56:24.919461 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:24.919359    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls podName:7bf35a3c-66f9-4ccf-b340-50a072ffa61f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:26.919349779 +0000 UTC m=+38.081069424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vqn7z" (UID: "7bf35a3c-66f9-4ccf-b340-50a072ffa61f") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:56:24.938791 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:24.938759    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:25.303630 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:25.303607    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9l9rr"]
Oct 31 00:56:25.318127 ip-10-0-139-174 kubenswrapper[2577]: W1031 00:56:25.318089    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ab3f3f_3f1d_4843_87ce_6b5a4e81ad1c.slice/crio-8d15149bb64daa496f016c41486394dc67686a01714fd6d72ee734e948de3e0d WatchSource:0}: Error finding container 8d15149bb64daa496f016c41486394dc67686a01714fd6d72ee734e948de3e0d: Status 404 returned error can't find the container with id 8d15149bb64daa496f016c41486394dc67686a01714fd6d72ee734e948de3e0d
Oct 31 00:56:25.516894 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:25.516831    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr" event={"ID":"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c","Type":"ContainerStarted","Data":"8d15149bb64daa496f016c41486394dc67686a01714fd6d72ee734e948de3e0d"}
Oct 31 00:56:25.893126 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:25.893087    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:26.522464 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:26.522423    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dljpn" event={"ID":"e39c2f85-6b3b-4343-bb84-c531243a36e5","Type":"ContainerStarted","Data":"50efadc183547ab5239a1609cdb909b1bf3fb7c810536a1a64058b08ae04ab2a"}
Oct 31 00:56:26.942111 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:26.942061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:26.942356 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:26.942216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:26.942356 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:26.942316    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:56:26.942536 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:26.942385    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm podName:7bf35a3c-66f9-4ccf-b340-50a072ffa61f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:30.942366239 +0000 UTC m=+42.104085907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vqn7z" (UID: "7bf35a3c-66f9-4ccf-b340-50a072ffa61f") : configmap "kube-rbac-proxy" not found
Oct 31 00:56:26.942619 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:26.942539    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:56:26.942619 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:26.942613    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls podName:7bf35a3c-66f9-4ccf-b340-50a072ffa61f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:30.942591782 +0000 UTC m=+42.104311440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vqn7z" (UID: "7bf35a3c-66f9-4ccf-b340-50a072ffa61f") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:56:27.841800 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:27.841699    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:28.530120 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:28.530088    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dsx4h-8b9gr" event={"ID":"506d546e-6187-4c6a-967b-09273e1fa785","Type":"ContainerStarted","Data":"2dd45acde478f2477a24827f5813867a2116a0de5904cf27d484b77616c415de"}
Oct 31 00:56:28.654051 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:28.653979    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dljpn" podStartSLOduration=4.547938084 podStartE2EDuration="8.653957547s" totalImagesPullingTime="4.106019463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:26.638470553 +0000 UTC m=+37.800190227" watchObservedRunningTime="2026-10-31 00:56:28.653957547 +0000 UTC m=+39.815677214"
Oct 31 00:56:28.654519 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:28.654478    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-dsx4h-8b9gr" podStartSLOduration=1.799783391 podStartE2EDuration="5.654466734s" totalImagesPullingTime="3.854683343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:28.654107329 +0000 UTC m=+39.815826985" watchObservedRunningTime="2026-10-31 00:56:28.654466734 +0000 UTC m=+39.816186394"
Oct 31 00:56:28.656438 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:28.656413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:28.656626 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:28.656571    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:56:28.656689 ip-10-0-139-174 kubenswrapper[2577]: E1031 00:56:28.656631    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls podName:f90cbecc-43bb-41a8-b743-4cdcd12dce2f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:36.656616546 +0000 UTC m=+47.818336185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls") pod "dns-default-5lczz" (UID: "f90cbecc-43bb-41a8-b743-4cdcd12dce2f") : secret "dns-default-metrics-tls" not found
Oct 31 00:56:29.117880 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:29.117839    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:29.490404 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:29.490319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:29.535296 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:29.535254    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qhvwh" event={"ID":"74673206-8b68-4414-8ac0-a63577cea44a","Type":"ContainerStarted","Data":"894d3f606566b6ad934e8bcf5a6326c0f3037faccc04c847645f2e60018fdea8"}
Oct 31 00:56:29.535469 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:29.535385    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:29.538009 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:29.537956    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr" event={"ID":"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c","Type":"ContainerStarted","Data":"6959b0db6a4f455d7e99f1799b49733cefe128d0e85e206d3aff80d2fcd58f99"}
Oct 31 00:56:29.651635 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:29.651581    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qhvwh" podStartSLOduration=34.878334327 podStartE2EDuration="39.651566993s" totalImagesPullingTime="4.773232666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:29.651261937 +0000 UTC m=+40.812981617" watchObservedRunningTime="2026-10-31 00:56:29.651566993 +0000 UTC m=+40.813286665"
Oct 31 00:56:29.720024 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:29.719994    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:29.765612 ip-10-0-139-174 kubenswrapper[2577]: I1031 00:56:29.765490    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr" podStartSLOduration=2.038054627 podStartE2EDuration="5.7654727s" totalImagesPullingTime="3.727418073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:29.765134241 +0000 UTC m=+40.926853906" watchObservedRunningTime="2026-10-31 00:56:29.7654727 +0000 UTC m=+40.927192364"
Oct 31 00:56:30.241928 ip-10-0-139-174 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 00:56:30.256665 ip-10-0-139-174 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 00:56:30.256905 ip-10-0-139-174 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 00:56:30.256971 ip-10-0-139-174 systemd[1]: kubelet.service: Consumed 2.478s CPU time.
Oct 31 00:56:30.267742 ip-10-0-139-174 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 00:56:30.364456 ip-10-0-139-174 kubenswrapper[4476]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:56:30.364456 ip-10-0-139-174 kubenswrapper[4476]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 00:56:30.364456 ip-10-0-139-174 kubenswrapper[4476]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:56:30.364456 ip-10-0-139-174 kubenswrapper[4476]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:56:30.367507 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367382    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:56:30.367507 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367505    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:56:30.367507 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367509    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:56:30.367507 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367512    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:56:30.367507 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367515    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367518    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367522    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367524    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367527    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367530    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367532    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367535    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367538    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367541    4476 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367543    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367547    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367549    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367552    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367555    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367557    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367560    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367562    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367565    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367568    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:56:30.367726 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367570    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367573    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367576    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367579    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367582    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367584    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367587    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367590    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367593    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367595    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367598    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367601    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367603    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367606    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367608    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367611    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367613    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367615    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367618    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367621    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:56:30.368241 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367623    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367625    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367628    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367630    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367633    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367636    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367639    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367642    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367644    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367647    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367650    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367652    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367654    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367657    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367659    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367662    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367665    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367667    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367670    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367672    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:56:30.368767 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367675    4476 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367678    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367680    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367684    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367686    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367689    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367692    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367694    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367697    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367699    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367702    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367704    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367707    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367710    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367713    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367715    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367718    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367721    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367723    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:56:30.369281 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367727    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367730    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367732    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367735    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367738    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367740    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367743    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367747    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367751    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367754    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367757    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367759    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367762    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367764    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367767    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367769    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367772    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367777    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367780    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:56:30.369760 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367783    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367786    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367788    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.367791    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368546    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368553    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368556    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368559    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368562    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368564    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368567    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368570    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368572    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368575    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368579    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368582    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368584    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368586    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368590    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:56:30.370229 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368594    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368598    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368603    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368608    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368611    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368614    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368616    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368618    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368621    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368623    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368626    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368628    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368631    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368633    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368636    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368638    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368641    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368644    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368647    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368650    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:56:30.370730 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368653    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368655    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368675    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368679    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368682    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368685    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368688    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368690    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368693    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368696    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368699    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368701    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368704    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368707    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368710    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368712    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368714    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368717    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368720    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368723    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:56:30.371284 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368725    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368728    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368730    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368733    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368736    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368738    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368741    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368743    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368745    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368747    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368750    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368752    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368755    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368757    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368759    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368762    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368765    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368768    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368770    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368773    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:56:30.371781 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368775    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368779    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368782    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368784    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368787    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368790    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368792    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368794    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368797    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368800    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368803    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368805    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368808    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368810    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368812    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368815    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368819    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368822    4476 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368826    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:56:30.372289 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368828    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368832    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368835    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368838    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368841    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368844    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368846    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368849    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368852    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368854    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368857    4476 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.368860    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369398    4476 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369404    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369407    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369410    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369414    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369416    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369419    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369422    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:56:30.372789 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369424    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369427    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369430    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369433    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369435    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369438    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369440    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369443    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369445    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369448    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369451    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369454    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369456    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369459    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369461    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369464    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369466    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369469    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369471    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369473    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:56:30.373301 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369476    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369478    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369481    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369483    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369486    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369491    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369494    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369497    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369508    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369511    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369514    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369516    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369519    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369521    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369525    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369528    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369531    4476 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369534    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369537    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:56:30.373795 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369540    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369543    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369547    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369550    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369553    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369555    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369558    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369561    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369564    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369566    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369569    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369572    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369574    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369577    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369579    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369582    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369584    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369587    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369591    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:56:30.374299 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369593    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369596    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369599    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369618    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369622    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369625    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369627    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369630    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369633    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369635    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369638    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369640    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369643    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369645    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369648    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369650    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369653    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369655    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369658    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369660    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:56:30.374782 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369664    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369666    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369669    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369671    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369674    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369676    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369679    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369681    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369683    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369686    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369688    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369692    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369695    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369697    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369700    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369703    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369706    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369708    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369711    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.369713    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369802    4476 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 00:56:30.375278 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369812    4476 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369821    4476 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369828    4476 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369834    4476 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369839    4476 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369846    4476 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369851    4476 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369854    4476 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369857    4476 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369861    4476 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369864    4476 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369867    4476 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369870    4476 flags.go:64] FLAG: --cgroup-root=""
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369873    4476 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369876    4476 flags.go:64] FLAG: --client-ca-file=""
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369879    4476 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369883    4476 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369887    4476 flags.go:64] FLAG: --cluster-domain=""
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369890    4476 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369893    4476 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369896    4476 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369900    4476 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369910    4476 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 00:56:30.375913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369913    4476 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369916    4476 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369919    4476 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369922    4476 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369926    4476 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369930    4476 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369933    4476 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369936    4476 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369940    4476 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369944    4476 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369946    4476 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369950    4476 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369953    4476 flags.go:64] FLAG: --enable-server="true"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369956    4476 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369961    4476 flags.go:64] FLAG: --event-burst="100"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369964    4476 flags.go:64] FLAG: --event-qps="50"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369967    4476 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369970    4476 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369974    4476 flags.go:64] FLAG: --eviction-hard=""
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369978    4476 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369981    4476 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369984    4476 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369987    4476 flags.go:64] FLAG: --eviction-soft=""
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369990    4476 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369993    4476 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 00:56:30.376484 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369996    4476 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.369999    4476 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370001    4476 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370004    4476 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370007    4476 flags.go:64] FLAG: --feature-gates=""
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370011    4476 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370016    4476 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370019    4476 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370022    4476 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370025    4476 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370028    4476 flags.go:64] FLAG: --help="false"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370031    4476 flags.go:64] FLAG: --hostname-override="ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370035    4476 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370038    4476 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370041    4476 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370044    4476 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370048    4476 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370050    4476 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370054    4476 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370056    4476 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370060    4476 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370063    4476 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370066    4476 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370069    4476 flags.go:64] FLAG: --kube-reserved=""
Oct 31 00:56:30.377168 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370072    4476 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370074    4476 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370077    4476 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370080    4476 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370083    4476 flags.go:64] FLAG: --lock-file=""
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370086    4476 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370088    4476 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370092    4476 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370097    4476 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370100    4476 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370103    4476 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370106    4476 flags.go:64] FLAG: --logging-format="text"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370109    4476 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370113    4476 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370117    4476 flags.go:64] FLAG: --manifest-url=""
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370119    4476 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370124    4476 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370127    4476 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370131    4476 flags.go:64] FLAG: --max-pods="110"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370134    4476 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370137    4476 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370141    4476 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370144    4476 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370147    4476 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370149    4476 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 00:56:30.377814 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370152    4476 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370160    4476 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370163    4476 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370166    4476 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370169    4476 flags.go:64] FLAG: --pod-cidr=""
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370188    4476 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370191    4476 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370194    4476 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370197    4476 flags.go:64] FLAG: --port="10250"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370200    4476 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370203    4476 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0fd33caf5398d3226"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370206    4476 flags.go:64] FLAG: --qos-reserved=""
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370210    4476 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370213    4476 flags.go:64] FLAG: --register-node="true"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370216    4476 flags.go:64] FLAG: --register-with-taints=""
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370220    4476 flags.go:64] FLAG: --registry-burst="10"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370223    4476 flags.go:64] FLAG: --registry-qps="5"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370225    4476 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370228    4476 flags.go:64] FLAG: --reserved-memory=""
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370234    4476 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370238    4476 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370242    4476 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370245    4476 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370248    4476 flags.go:64] FLAG: --runonce="false"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370251    4476 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370254    4476 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370257    4476 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 00:56:30.378448 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370260    4476 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370263    4476 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370266    4476 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370269    4476 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370273    4476 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370276    4476 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370278    4476 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370281    4476 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370284    4476 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370287    4476 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370290    4476 flags.go:64] FLAG: --system-cgroups=""
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370293    4476 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370301    4476 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370305    4476 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370309    4476 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370313    4476 flags.go:64] FLAG: --tls-min-version=""
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370316    4476 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370319    4476 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370321    4476 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370324    4476 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370327    4476 flags.go:64] FLAG: --v="2"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370332    4476 flags.go:64] FLAG: --version="false"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370336    4476 flags.go:64] FLAG: --vmodule=""
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370340    4476 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370344    4476 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 00:56:30.379104 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370488    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370493    4476 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370497    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370499    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370502    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370505    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370507    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370510    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370512    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370515    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370518    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370521    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370524    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370526    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370529    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370531    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370534    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370536    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370539    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:56:30.380111 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370541    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370544    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370547    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370550    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370552    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370555    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370558    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370560    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370562    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370565    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370567    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370570    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370573    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370576    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370579    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370582    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370584    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370587    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370589    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370592    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:56:30.380624 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370594    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370597    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370599    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370602    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370605    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370607    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370610    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370612    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370615    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370617    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370620    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370623    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370627    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370630    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370633    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370635    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370638    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370640    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370642    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:56:30.381169 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370645    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370647    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370650    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370652    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370655    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370658    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370661    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370665    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370668    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370670    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370673    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370675    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370678    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370680    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370683    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370685    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370688    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370690    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370694    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370696    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:56:30.381851 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370699    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370701    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370704    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370706    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370708    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370711    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370714    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370716    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370719    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370721    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370723    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370726    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370728    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370731    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370733    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370736    4476 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370739    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370743    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370746    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370750    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:56:30.382860 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370753    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370755    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370758    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370760    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370762    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370765    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370769    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370772    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.370799    4476 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370836    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370840    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370844    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370847    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370849    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370852    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370855    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:56:30.383853 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370858    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370860    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370863    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370866    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370868    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370871    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370873    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370876    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370878    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370881    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370883    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370886    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370888    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370892    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370894    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370897    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370900    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370903    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370905    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370908    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:56:30.384741 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370910    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370913    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370915    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370918    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370920    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370922    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370925    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370927    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370935    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370940    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370944    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370947    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370950    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370952    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370955    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370957    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370959    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370962    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370964    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:56:30.385674 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370967    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370969    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370972    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370974    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370977    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370979    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370982    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370984    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370987    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370990    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370993    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370995    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.370998    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371000    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371003    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371006    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371008    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371011    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371013    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371016    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:56:30.386303 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371018    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371021    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371024    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371027    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371029    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371032    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371034    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371036    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371039    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371041    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371044    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371046    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371049    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371051    4476 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371054    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371056    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371059    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371061    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371064    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371066    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:56:30.386809 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371068    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371072    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371074    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371077    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371081    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371084    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371087    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371089    4476 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371092    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371094    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371096    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371099    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371101    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371104    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371106    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371109    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371112    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371114    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371117    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:56:30.387508 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.371119    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.375129    4476 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.375144    4476 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.375163    4476 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.375168    4476 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375253    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375258    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375261    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375263    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375266    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375269    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375271    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375274    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375277    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375280    4476 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375283    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375285    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375288    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375291    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:56:30.388068 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375293    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375296    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375300    4476 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375302    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375305    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375307    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375310    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375312    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375315    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375325    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375328    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375330    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375333    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375335    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375338    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375340    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375343    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375345    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375348    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375350    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:56:30.388769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375353    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375355    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375358    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375360    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375362    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375365    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375369    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375373    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375377    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375379    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375382    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375385    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375388    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375391    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375393    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375396    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375398    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375401    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375403    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:56:30.389275 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375406    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375409    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375411    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375414    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375417    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375419    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375422    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375424    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375426    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375429    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375431    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375434    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375436    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375438    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375441    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375443    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375446    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375449    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375451    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:56:30.389769 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375454    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375457    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375459    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375462    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375464    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375468    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375470    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375472    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375475    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375477    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375480    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375482    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375485    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375487    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375490    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375492    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375495    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375498    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375501    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375505    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:56:30.390264 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375508    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375511    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375514    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375516    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375519    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375522    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375524    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375527    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375529    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375532    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375534    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375537    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375539    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375542    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.375570    4476 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375737    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:56:30.390917 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375742    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375745    4476 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375748    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375750    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375752    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375755    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375757    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375760    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375763    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375765    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375768    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375770    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375773    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375775    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375778    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375781    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375783    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375786    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375788    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375791    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:56:30.391593 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375793    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375795    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375798    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375801    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375803    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375805    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375808    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375811    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375813    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375816    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375819    4476 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375822    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375824    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375827    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375829    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375832    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375834    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375837    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375839    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:56:30.392262 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375842    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375844    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375847    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375849    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375852    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375854    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375857    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375859    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375862    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375865    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375867    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375870    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375872    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375875    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375877    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375879    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375882    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375884    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375887    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375889    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:56:30.392732 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375892    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375894    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375897    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375900    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375902    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375904    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375908    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375912    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375915    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375918    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375921    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375924    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375930    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375934    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375937    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375940    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375942    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375944    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375947    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:56:30.393271 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375949    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375952    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375955    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375957    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375960    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375962    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375964    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375967    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375970    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375972    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375976    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375979    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375981    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375984    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375986    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375988    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375991    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375993    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375996    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.375998    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:56:30.393757 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376001    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376004    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376006    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376009    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376011    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376015    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376017    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.376040    4476 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376089    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376093    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376096    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376099    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376102    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376104    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376107    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376110    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:56:30.394322 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376113    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376116    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376120    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376123    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376126    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376129    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376132    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376138    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376141    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376144    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376147    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376149    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376152    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376155    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376158    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376161    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376163    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376166    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376168    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:56:30.394746 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376184    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376187    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376191    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376193    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376196    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376199    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376201    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376204    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376207    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376209    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376212    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376214    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376217    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376220    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376223    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376225    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376228    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376231    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376233    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:56:30.395214 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376236    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376240    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376243    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376246    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376248    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376251    4476 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376253    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376256    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376259    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376261    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376264    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376266    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376269    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376272    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376274    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376279    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376282    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376284    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376287    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376289    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:56:30.395682 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376292    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376294    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376297    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376299    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376302    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376304    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376307    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376316    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376318    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376321    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376324    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376326    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376328    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376332    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376335    4476 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376337    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376340    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376343    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376345    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:56:30.396354 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376348    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376350    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376353    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376355    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376358    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376360    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376363    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376365    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376369    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376371    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376374    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376376    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376383    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376386    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376388    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376391    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376393    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376395    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376398    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376400    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:56:30.397126 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:30.376402    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:56:30.397902 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.376457    4476 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 00:56:30.397902 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.377701    4476 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 00:56:30.397902 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.377760    4476 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 00:56:30.397902 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.377941    4476 server.go:1014] "Starting client certificate rotation"
Oct 31 00:56:30.397902 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.378099    4476 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:56:30.397902 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.378216    4476 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:50:50 +0000 UTC" deadline="2028-05-23 09:48:29.662979321 +0000 UTC"
Oct 31 00:56:30.397902 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.378289    4476 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13688h51m59.284694065s"
Oct 31 00:56:30.397902 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.378613    4476 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:56:30.397902 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.380189    4476 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:56:30.397902 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.383574    4476 log.go:25] "Validated CRI v1 runtime API"
Oct 31 00:56:30.397902 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.387140    4476 log.go:25] "Validated CRI v1 image API"
Oct 31 00:56:30.397902 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.388034    4476 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 00:56:30.397902 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.389642    4476 fs.go:135] Filesystem UUIDs: map[07052935-016b-4273-8522-16f400d4650f:/dev/nvme0n1p3 3826533a-182a-4dab-8b4b-22e346293988:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 31 00:56:30.398509 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.389665    4476 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31d5f54f34e5d2a974970231e806aa8341c2b7b385a44854d947bf56215d519f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31d5f54f34e5d2a974970231e806aa8341c2b7b385a44854d947bf56215d519f/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35e2c18924ca3bf73037fd4b2b57bcd54be0ecc893851fa01325cdef2cd9ffa2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35e2c18924ca3bf73037fd4b2b57bcd54be0ecc893851fa01325cdef2cd9ffa2/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/490615ce9201733ff2579561f1eca8528fda8376df97bab8681e5b92a8143865/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/490615ce9201733ff2579561f1eca8528fda8376df97bab8681e5b92a8143865/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e45f6dff91a07a8b722c2deb45bb14c00b1e539af4914ad78a6d6c1b93bdb66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e45f6dff91a07a8b722c2deb45bb14c00b1e539af4914ad78a6d6c1b93bdb66/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a2a1f63e3e81d3ccdbdd2c6456fc992387964f779731a8c8285a71d41e76f9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a2a1f63e3e81d3ccdbdd2c6456fc992387964f779731a8c8285a71d41e76f9e/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d15149bb64daa496f016c41486394dc67686a01714fd6d72ee734e948de3e0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d15149bb64daa496f016c41486394dc67686a01714fd6d72ee734e948de3e0d/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aaf8df7b834dd3e771643f1bc0143c52b7f25f9bcfb196d2c85069811d3ca94f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aaf8df7b834dd3e771643f1bc0143c52b7f25f9bcfb196d2c85069811d3ca94f/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af00f6f9ab3bc53a0cbf6eb4a44ea51d728c9ffddc86ddfbc6817d7c46124b70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af00f6f9ab3bc53a0cbf6eb4a44ea51d728c9ffddc86ddfbc6817d7c46124b70/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2bba292241a8498175712a6eda34cd0491c8ef41e87e0a45f7425d8fdce9ff1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2bba292241a8498175712a6eda34cd0491c8ef41e87e0a45f7425d8fdce9ff1/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4459dedf25045e99b03c872e7883f37e2c7ddef022171c54d199e6d20e75bab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4459dedf25045e99b03c872e7883f37e2c7ddef022171c54d199e6d20e75bab/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfbbc42e46c5051dc778989248e094d6ffeb4f61582bc2f5d423a50012d9acc4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfbbc42e46c5051dc778989248e094d6ffeb4f61582bc2f5d423a50012d9acc4/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e05e91dc951626d118ea0c64ef66f49a8b3a1e997129a15262652912c4d78815/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e05e91dc951626d118ea0c64ef66f49a8b3a1e997129a15262652912c4d78815/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e815def07a2cef6fc6b1bb2468d81241584a117d5ed3b335cd1611d5886f2903/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e815def07a2cef6fc6b1bb2468d81241584a117d5ed3b335cd1611d5886f2903/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2712ec09b95f839afc49b8f58afa9ff40d6674dac5109fb129f5c4946003e54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2712ec09b95f839afc49b8f58afa9ff40d6674dac5109fb129f5c4946003e54/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff6b779da7560ae07af4993b0a32ba95bb04a57de51482b122b2a7f82f4c173c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff6b779da7560ae07af4993b0a32ba95bb04a57de51482b122b2a7f82f4c173c/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12564994-4d47-4401-bc99-a4755b7ed33f/volumes/kubernetes.io~projected/kube-api-access-h86lw:{mountpoint:/var/lib/kubelet/pods/12564994-4d47-4401-bc99-a4755b7ed33f/volumes/kubernetes.io~projected/kube-api-access-h86lw major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27c82039-550c-4fe8-96c7-dfd8990440e4/volumes/kubernetes.io~projected/kube-api-access-7rz6k:{mountpoint:/var/lib/kubelet/pods/27c82039-550c-4fe8-96c7-dfd8990440e4/volumes/kubernetes.io~projected/kube-api-access-7rz6k major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c/volumes/kubernetes.io~projected/kube-api-access-4wd9l:{mountpoint:/var/lib/kubelet/pods/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c/volumes/kubernetes.io~projected/kube-api-access-4wd9l major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c/volumes/kubernetes.io~secret/signing-key major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/506d546e-6187-4c6a-967b-09273e1fa785/volumes/kubernetes.io~projected/kube-api-access-nrm7b:{mountpoint:/var/lib/kubelet/pods/506d546e-6187-4c6a-967b-09273e1fa785/volumes/kubernetes.io~projected/kube-api-access-nrm7b major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68af8611-a641-4663-b13b-8f015107e82f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/68af8611-a641-4663-b13b-8f015107e82f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68af8611-a641-4663-b13b-8f015107e82f/volumes/kubernetes.io~projected/kube-api-access-kfxdb:{mountpoint:/var/lib/kubelet/pods/68af8611-a641-4663-b13b-8f015107e82f/volumes/kubernetes.io~projected/kube-api-access-kfxdb major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68af8611-a641-4663-b13b-8f015107e82f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/68af8611-a641-4663-b13b-8f015107e82f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74673206-8b68-4414-8ac0-a63577cea44a/volumes/kubernetes.io~projected/kube-api-access-5pdzc:{mountpoint:/var/lib/kubelet/pods/74673206-8b68-4414-8ac0-a63577cea44a/volumes/kubernetes.io~projected/kube-api-access-5pdzc major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ae98732-67f4-420b-9e5b-e4e311ac9f9c/volumes/kubernetes.io~projected/kube-api-access-2vwtr:{mountpoint:/var/lib/kubelet/pods/7ae98732-67f4-420b-9e5b-e4e311ac9f9c/volumes/kubernetes.io~projected/kube-api-access-2vwtr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b893def-2fc6-4350-a01f-6ee4969c48bd/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7b893def-2fc6-4350-a01f-6ee4969c48bd/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bf35a3c-66f9-4ccf-b340-50a072ffa61f/volumes/kubernetes.io~projected/kube-api-access-47hxq:{mountpoint:/var/lib/kubelet/pods/7bf35a3c-66f9-4ccf-b340-50a072ffa61f/volumes/kubernetes.io~projected/kube-api-access-47hxq major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bd7a453-d0b1-4db7-8dfa-247a72467481/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9bd7a453-d0b1-4db7-8dfa-247a72467481/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bd7a453-d0b1-4db7-8dfa-247a72467481/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9bd7a453-d0b1-4db7-8dfa-247a72467481/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bd7a453-d0b1-4db7-8dfa-247a72467481/volumes/kubernetes.io~projected/kube-api-access-q9mcc:{mountpoint:/var/lib/kubelet/pods/9bd7a453-d0b1-4db7-8dfa-247a72467481/volumes/kubernetes.io~projected/kube-api-access-q9mcc major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d67aaf2-9211-4f6c-84d8-99aae85c28eb/volumes/kubernetes.io~projected/kube-api-access-542pt:{mountpoint:/var/lib/kubelet/pods/9d67aaf2-9211-4f6c-84d8-99aae85c28eb/volumes/kubernetes.io~projected/kube-api-access-542pt major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac61d127-75ba-4f73-bcc6-7d69bf320a55/volumes/kubernetes.io~projected/kube-api-access-vr7vl:{mountpoint:/var/lib/kubelet/pods/ac61d127-75ba-4f73-bcc6-7d69bf320a55/volumes/kubernetes.io~projected/kube-api-access-vr7vl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0cade88-4da4-467c-9656-cce6afe1dd62/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e0cade88-4da4-467c-9656-cce6afe1dd62/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e39c2f85-6b3b-4343-bb84-c531243a36e5/volumes/kubernetes.io~projected/kube-api-access-jdcx7:{mountpoint:/var/lib/kubelet/pods/e39c2f85-6b3b-4343-bb84-c531243a36e5/volumes/kubernetes.io~projected/kube-api-access-jdcx7 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e39c2f85-6b3b-4343-bb84-c531243a36e5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e39c2f85-6b3b-4343-bb84-c531243a36e5/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4311d66-bb16-4a76-a021-527f4c1395f4/volumes/kubernetes.io~projected/kube-api-access-t522b:{mountpoint:/var/lib/kubelet/pods/e4311d66-bb16-4a76-a021-527f4c1395f4/volumes/kubernetes.io~projected/kube-api-access-t522b major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f21550fa-496b-4806-abd8-7f0441da9256/volumes/kubernetes.io~projected/kube-api-access-cphjb:{mountpoint:/var/lib/kubelet/pods/f21550fa-496b-4806-abd8-7f0441da9256/volumes/kubernetes.io~projected/kube-api-access-cphjb major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f90cbecc-43bb-41a8-b743-4cdcd12dce2f/volumes/kubernetes.io~projected/kube-api-access-shw5k:{mountpoint:/var/lib/kubelet/pods/f90cbecc-43bb-41a8-b743-4cdcd12dce2f/volumes/kubernetes.io~projected/kube-api-access-shw5k major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/132503d5ae6c7924652b71176e8599dcf6fb2d86c84ab1cab9035d5a57f8b661/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/af409774245f4eca655d07cc3589f52052029abf9a11b3b230ec30757ebabf22/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/cf4b5c28d43abf0eadb61f028a38589460241c7ee461642019738032499dd523/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/25a3ab4389b57e9f23d5567bd68b14ac0e68bf9876ba37a00a57848090caefde/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/5002733b7187f82426509af5c7a98861f6f9d13684c7a0bbe9057c75ba5d1053/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/59d7c0ae3f25a3ddf973e38abf5eac1591754d9e64df8115cc6b6cb07e8b55af/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/be9bb82f4617dd1c4fa909de24e33b2470d1c3c489478ed17125f29af62a77b4/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/782a0792c14c2ecb5b56b5ae24feba6c06b13de904d30705edaa50eafa73c40a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/fc7f7ea98512e3cfe0197d433bfa22fd982576575c3d90e465468475f1e436c4/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/011ba20c56a6bbd3bf24b737b5ba2e3f2b3ffcd97e938db55f0d5ac71148bfbe/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/ecb885fcc801673e925249bd3b544937c132a74aee6fab59c3713506145a9ddb/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/aabec11edf02162bee1bd79501dcd45d53d516b206fc26502c5bb66c30654364/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/259fda0a7faaeaef99bf79cf2c3c0ae8a4b21f24d380288d2930eb5a9a7ad1cd/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b6ccd1f9ea0cc6f23ce2a596293f2e050be99481a2e9242fda700ca893cab110/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/78b7609086b381ab6dbf62d0335166add36eeaf3085dbf1322741e87f2f16410/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/aebb54a291b26b9c4249d305ef53e1a6446c9456dfbf47240ee91113e18b2d43/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/12aaeec520b42bea8fc8693220f44b55f4737d8ae0d8ddc9db54525961b0e335/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/56903d5689ed04d7f609609dc9377f7e37e023df863f2a5f355952119e65c2b0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a7debb7acf65c73923068aadb9ee332a76115cb2d419fde0a68f2737fef3200a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/603cc3591b5c81897cf19d1f751ee2af6df58e10de4728fdffb0e4b36a303447/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7eb9f7ee6b59a6cbb1ae53c9b92ca9a625f90a68552d0f77e4ba32b4af40d01d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/51142b4b2c8a0ba6fe3e50c9f4f635a3303f00f0d0e09d6df06c55c0cdf681d5/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/e677652681b21aa2610b125864c463ad0c66d475b07e92ffd5577530541feeba/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/60dae5f973f3bbe7304b662d87605b65002e560224646851e62f73fcdff76944/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/9d5f85dde6b4d88071ce3de0c87f75f4332a8afb7679955e0e3fd47340460746/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/4caf8d9ddd6ff4da50596da4b2c0ac37e187564c8204462213541225e7ba4798/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/3fb7519189998e4b3ea96b8f7a47b7097ac3ff8408ea5242701bf5f2832a78c1/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/542d1521611b7ea5eaf727f8a10aeda8fa6176043228ab9db6e2f57d04dcd2db/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/50bb65f269a59c30450c62b5f2e734d6c7a8331817b388f9de762812bf8ada61/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/ded3d820b66b28f1d43907a1a4a84bcca04f71f37e58408b4e7fa08eeec7a862/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/981b654c688075de917e3349eb0dc6e0b415b94aee06d82ae2f81bedb718bd1b/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b91cbd90de2e6394741e08ad1215bdb2edc02f199868c358b4ded374632c1969/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/7455d648c6ce45f9e3909dc7ef44e75cc57441b15d1a2da8d393de4c1b0e3371/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1ea75e4bbffae656221655fead2c96c85f410ab191fbe4e6b8f627064f90af5e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/13d3b02a42a66bb24d19bb535f8e8e308200d16b8b6d80a256b4ede1451ced67/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/78f92d6ab810920e14591cbcc57293682fabf935f8e0a619bf3b2f4e35abc15e/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/ae4111d05c4ea5cbf6147279066204442eb897ece2bc285825d3b81ccde84509/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/3b0a830bd3287a186bf8a22dafa2cdcd008a21276c38b90d492b633a23e6767a/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/c088b896e244717760e410cde88436265a8f653e38d12235c5da1db4fa5fc3b3/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8105a7c378f25f0e95876106036188d278aee5092ce0aecf706d006e2458d2e7/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/49db8157c088c09c19085b04572e1b756d9e6ee87f9a58e8e84b6186aebc6ef8/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.395674    4476 manager.go:217] Machine: {Timestamp:2026-10-31 00:56:30.395202663 +0000 UTC m=+0.085085781 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3051121 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24c640e66eb85dc6d742837fe9cd36 SystemUUID:ec24c640-e66e-b85d-c6d7-42837fe9cd36 BootID:b83be239-d4b1-4f24-99b7-51d2acc4a43e Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/e4311d66-bb16-4a76-a021-527f4c1395f4/volumes/kubernetes.io~projected/kube-api-access-t522b DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfbbc42e46c5051dc778989248e094d6ffeb4f61582bc2f5d423a50012d9acc4/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f21550fa-496b-4806-abd8-7f0441da9256/volumes/kubernetes.io~projected/kube-api-access-cphjb DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2bba292241a8498175712a6eda34cd0491c8ef41e87e0a45f7425d8fdce9ff1/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27c82039-550c-4fe8-96c7-dfd8990440e4/volumes/kubernetes.io~projected/kube-api-access-7rz6k DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/12564994-4d47-4401-bc99-a4755b7ed33f/volumes/kubernetes.io~projected/kube-api-access-h86lw DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/68af8611-a641-4663-b13b-8f015107e82f/volumes/kubernetes.io~projected/kube-api-access-kfxdb DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e45f6dff91a07a8b722c2deb45bb14c00b1e539af4914ad78a6d6c1b93bdb66/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/74673206-8b68-4414-8ac0-a63577cea44a/volumes/kubernetes.io~projected/kube-api-access-5pdzc DeviceMajor:0 DeviceMinor:209 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d15149bb64daa496f016c41486394dc67686a01714fd6d72ee734e948de3e0d/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bd7a453-d0b1-4db7-8dfa-247a72467481/volumes/kubernetes.io~projected/kube-api-access-q9mcc DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af00f6f9ab3bc53a0cbf6eb4a44ea51d728c9ffddc86ddfbc6817d7c46124b70/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2712ec09b95f839afc49b8f58afa9ff40d6674dac5109fb129f5c4946003e54/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aaf8df7b834dd3e771643f1bc0143c52b7f25f9bcfb196d2c85069811d3ca94f/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/68af8611-a641-4663-b13b-8f015107e82f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/490615ce9201733ff2579561f1eca8528fda8376df97bab8681e5b92a8143865/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ae98732-67f4-420b-9e5b-e4e311ac9f9c/volumes/kubernetes.io~projected/kube-api-access-2vwtr DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a2a1f63e3e81d3ccdbdd2c6456fc992387964f779731a8c8285a71d41e76f9e/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c/volumes/kubernetes.io~projected/kube-api-access-4wd9l DeviceMajor:0 DeviceMinor:223 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31d5f54f34e5d2a974970231e806aa8341c2b7b385a44854d947bf56215d519f/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bd7a453-d0b1-4db7-8dfa-247a72467481/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e815def07a2cef6fc6b1bb2468d81241584a117d5ed3b335cd1611d5886f2903/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e05e91dc951626d118ea0c64ef66f49a8b3a1e997129a15262652912c4d78815/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e39c2f85-6b3b-4343-bb84-c531243a36e5/volumes/kubernetes.io~projected/kube-api-access-jdcx7 DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e39c2f85-6b3b-4343-bb84-c531243a36e5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f90cbecc-43bb-41a8-b743-4cdcd12dce2f/volumes/kubernetes.io~projected/kube-api-access-shw5k DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35e2c18924ca3bf73037fd4b2b57bcd54be0ecc893851fa01325cdef2cd9ffa2/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff6b779da7560ae07af4993b0a32ba95bb04a57de51482b122b2a7f82f4c173c/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68af8611-a641-4663-b13b-8f015107e82f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/506d546e-6187-4c6a-967b-09273e1fa785/volumes/kubernetes.io~projected/kube-api-access-nrm7b DeviceMajor:0 DeviceMinor:215 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/7b893def-2fc6-4350-a01f-6ee4969c48bd/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9d67aaf2-9211-4f6c-84d8-99aae85c28eb/volumes/kubernetes.io~projected/kube-api-access-542pt DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:222 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/ac61d127-75ba-4f73-bcc6-7d69bf320a55/volumes/kubernetes.io~projected/kube-api-access-vr7vl DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7bf35a3c-66f9-4ccf-b340-50a072ffa61f/volumes/kubernetes.io~projected/kube-api-access-47hxq DeviceMajor:0 DeviceMinor:210 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4459dedf25045e99b03c872e7883f37e2c7ddef022171c54d199e6d20e75bab/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bd7a453-d0b1-4db7-8dfa-247a72467481/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0cade88-4da4-467c-9656-cce6afe1dd62/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:31d5f54f34e5d2a MacAddress:86:88:e6:14:db:e5 Speed:10000 Mtu:8901} {Name:35e2c18924ca3bf MacAddress:aa:46:23:24:f2:92 Speed:10000 Mtu:8901} {Name:6e45f6dff91a07a MacAddress:82:92:ee:22:7b:04 Speed:10000 Mtu:8901} {Name:8d15149bb64daa4 MacAddress:d2:f3:d3:38:e6:c5 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:24:40:e3:23:97 Speed:0 Mtu:9001} {Name:br-int MacAddress:1a:d7:3e:8d:36:62 Speed:0 Mtu:8901} {Name:e815def07a2cef6 MacAddress:d6:27:50:1a:ea:2a Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:24:40:e3:23:97 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:96:08:24:04:2e:8b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:12:7f:4d:94:40:aa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.395920    4476 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.395963    4476 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.396337    4476 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.396365    4476 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-174.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.396562    4476 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.396574    4476 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.396589    4476 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.396611    4476 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.396753    4476 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.397002    4476 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.397139    4476 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.397206    4476 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.397226    4476 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.397240    4476 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.397255    4476 kubelet.go:405] "Adding apiserver pod source"
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.397268    4476 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.398267    4476 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 00:56:30.398704 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.398280    4476 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:56:30.399402 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.398834    4476 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 00:56:30.400888 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.400871    4476 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 00:56:30.400888 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.400888    4476 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:56:30.401000 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.400990    4476 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 00:56:30.401228 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401216    4476 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 00:56:30.401273 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401247    4476 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 00:56:30.401359 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401349    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 00:56:30.401388 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401366    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 00:56:30.401388 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401372    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 00:56:30.401388 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401378    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 00:56:30.401388 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401384    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 00:56:30.401493 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401390    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 00:56:30.401493 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401396    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 00:56:30.401493 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401401    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 00:56:30.401493 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401408    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 00:56:30.401493 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401415    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 00:56:30.401493 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401428    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 00:56:30.401493 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401438    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 00:56:30.401493 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401464    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 00:56:30.401493 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.401470    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 00:56:30.404933 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.404920    4476 server.go:1263] "Started kubelet"
Oct 31 00:56:30.405853 ip-10-0-139-174 systemd[1]: Started Kubernetes Kubelet.
Oct 31 00:56:30.407963 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.405046    4476 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 00:56:30.407963 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.405467    4476 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 00:56:30.407963 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.407097    4476 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 00:56:30.407963 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.407451    4476 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 00:56:30.407963 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.407540    4476 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 00:56:30.410157 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.410139    4476 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 00:56:30.410458 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.410209    4476 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 00:56:30.410535 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.410446    4476 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 00:56:30.410535 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.410437    4476 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 00:56:30.410535 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.410488    4476 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 00:56:30.410693 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.410601    4476 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:50:52 +0000 UTC" deadline="2028-04-21 11:47:08.008668087 +0000 UTC"
Oct 31 00:56:30.410693 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.410622    4476 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12922h50m37.598049701s"
Oct 31 00:56:30.411032 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:30.410988    4476 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-174.us-west-1.compute.internal\" not found"
Oct 31 00:56:30.411909 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.411892    4476 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 00:56:30.412025 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412014    4476 factory.go:55] Registering systemd factory
Oct 31 00:56:30.412143 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412131    4476 factory.go:223] Registration of the systemd container factory successfully
Oct 31 00:56:30.412529 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412513    4476 factory.go:153] Registering CRI-O factory
Oct 31 00:56:30.412601 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412533    4476 factory.go:223] Registration of the crio container factory successfully
Oct 31 00:56:30.412601 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412554    4476 factory.go:103] Registering Raw factory
Oct 31 00:56:30.412601 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412569    4476 manager.go:1199] Started watching for new ooms in manager
Oct 31 00:56:30.412601 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412505    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12564994-4d47-4401-bc99-a4755b7ed33f" volumeName="kubernetes.io/configmap/12564994-4d47-4401-bc99-a4755b7ed33f-cni-binary-copy" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412608    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c" volumeName="kubernetes.io/projected/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-kube-api-access-4wd9l" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412623    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68af8611-a641-4663-b13b-8f015107e82f" volumeName="kubernetes.io/projected/68af8611-a641-4663-b13b-8f015107e82f-kube-api-access-kfxdb" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412637    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b893def-2fc6-4350-a01f-6ee4969c48bd" volumeName="kubernetes.io/secret/7b893def-2fc6-4350-a01f-6ee4969c48bd-agent-certs" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412649    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4311d66-bb16-4a76-a021-527f4c1395f4" volumeName="kubernetes.io/projected/e4311d66-bb16-4a76-a021-527f4c1395f4-kube-api-access-t522b" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412661    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f21550fa-496b-4806-abd8-7f0441da9256" volumeName="kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412674    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68af8611-a641-4663-b13b-8f015107e82f" volumeName="kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-env-overrides" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412685    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68af8611-a641-4663-b13b-8f015107e82f" volumeName="kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412700    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f21550fa-496b-4806-abd8-7f0441da9256" volumeName="kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-cni-binary-copy" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412713    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="506d546e-6187-4c6a-967b-09273e1fa785" volumeName="kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412725    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ae98732-67f4-420b-9e5b-e4e311ac9f9c" volumeName="kubernetes.io/projected/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-kube-api-access-2vwtr" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412739    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bd7a453-d0b1-4db7-8dfa-247a72467481" volumeName="kubernetes.io/projected/9bd7a453-d0b1-4db7-8dfa-247a72467481-kube-api-access-q9mcc" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412753    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0cade88-4da4-467c-9656-cce6afe1dd62" volumeName="kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412765    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f21550fa-496b-4806-abd8-7f0441da9256" volumeName="kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412778    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f21550fa-496b-4806-abd8-7f0441da9256" volumeName="kubernetes.io/projected/f21550fa-496b-4806-abd8-7f0441da9256-kube-api-access-cphjb" seLinuxMountContext=""
Oct 31 00:56:30.412786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412789    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d67aaf2-9211-4f6c-84d8-99aae85c28eb" volumeName="kubernetes.io/projected/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-kube-api-access-542pt" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412801    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12564994-4d47-4401-bc99-a4755b7ed33f" volumeName="kubernetes.io/configmap/12564994-4d47-4401-bc99-a4755b7ed33f-multus-daemon-config" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412813    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27c82039-550c-4fe8-96c7-dfd8990440e4" volumeName="kubernetes.io/projected/27c82039-550c-4fe8-96c7-dfd8990440e4-kube-api-access-7rz6k" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412826    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="506d546e-6187-4c6a-967b-09273e1fa785" volumeName="kubernetes.io/empty-dir/506d546e-6187-4c6a-967b-09273e1fa785-archives-path" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412839    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68af8611-a641-4663-b13b-8f015107e82f" volumeName="kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-ovnkube-config" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412851    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bf35a3c-66f9-4ccf-b340-50a072ffa61f" volumeName="kubernetes.io/empty-dir/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-data-volume" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412862    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac61d127-75ba-4f73-bcc6-7d69bf320a55" volumeName="kubernetes.io/projected/ac61d127-75ba-4f73-bcc6-7d69bf320a55-kube-api-access-vr7vl" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412873    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f90cbecc-43bb-41a8-b743-4cdcd12dce2f" volumeName="kubernetes.io/empty-dir/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-tmp-dir" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412884    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bd7a453-d0b1-4db7-8dfa-247a72467481" volumeName="kubernetes.io/empty-dir/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-tuned" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412895    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c" volumeName="kubernetes.io/configmap/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-signing-cabundle" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412907    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="506d546e-6187-4c6a-967b-09273e1fa785" volumeName="kubernetes.io/projected/506d546e-6187-4c6a-967b-09273e1fa785-kube-api-access-nrm7b" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412920    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74673206-8b68-4414-8ac0-a63577cea44a" volumeName="kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412931    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b893def-2fc6-4350-a01f-6ee4969c48bd" volumeName="kubernetes.io/configmap/7b893def-2fc6-4350-a01f-6ee4969c48bd-konnectivity-ca" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412960    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e39c2f85-6b3b-4343-bb84-c531243a36e5" volumeName="kubernetes.io/secret/e39c2f85-6b3b-4343-bb84-c531243a36e5-cert" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412972    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f90cbecc-43bb-41a8-b743-4cdcd12dce2f" volumeName="kubernetes.io/configmap/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-config-volume" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412984    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="506d546e-6187-4c6a-967b-09273e1fa785" volumeName="kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-service-ca-bundle" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412995    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac61d127-75ba-4f73-bcc6-7d69bf320a55" volumeName="kubernetes.io/configmap/ac61d127-75ba-4f73-bcc6-7d69bf320a55-iptables-alerter-script" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.413006    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e39c2f85-6b3b-4343-bb84-c531243a36e5" volumeName="kubernetes.io/projected/e39c2f85-6b3b-4343-bb84-c531243a36e5-kube-api-access-jdcx7" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.413018    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4311d66-bb16-4a76-a021-527f4c1395f4" volumeName="kubernetes.io/configmap/e4311d66-bb16-4a76-a021-527f4c1395f4-serviceca" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.413030    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12564994-4d47-4401-bc99-a4755b7ed33f" volumeName="kubernetes.io/projected/12564994-4d47-4401-bc99-a4755b7ed33f-kube-api-access-h86lw" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.413041    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f90cbecc-43bb-41a8-b743-4cdcd12dce2f" volumeName="kubernetes.io/projected/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-kube-api-access-shw5k" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.413053    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bf35a3c-66f9-4ccf-b340-50a072ffa61f" volumeName="kubernetes.io/projected/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-api-access-47hxq" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.413064    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bd7a453-d0b1-4db7-8dfa-247a72467481" volumeName="kubernetes.io/empty-dir/9bd7a453-d0b1-4db7-8dfa-247a72467481-tmp" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.413076    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c" volumeName="kubernetes.io/secret/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-signing-key" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.413087    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68af8611-a641-4663-b13b-8f015107e82f" volumeName="kubernetes.io/secret/68af8611-a641-4663-b13b-8f015107e82f-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.413099    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ae98732-67f4-420b-9e5b-e4e311ac9f9c" volumeName="kubernetes.io/empty-dir/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-tmp-dir" seLinuxMountContext=""
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.413111    4476 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.413119    4476 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 00:56:30.413546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.412912    4476 manager.go:319] Starting recovery of all containers
Oct 31 00:56:30.445840 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.444657    4476 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 00:56:30.446421 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.446392    4476 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 00:56:30.446541 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.446426    4476 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 00:56:30.446541 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.446450    4476 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 00:56:30.446541 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:30.446495    4476 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 00:56:30.457244 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.457216    4476 generic.go:356] "Generic (PLEG): container finished" podID="f21550fa-496b-4806-abd8-7f0441da9256" containerID="724b362628522850930523214c51dcab8af71b86684609170c6c5bf481af5cd9" exitCode=0
Oct 31 00:56:30.457244 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.457238    4476 generic.go:356] "Generic (PLEG): container finished" podID="f21550fa-496b-4806-abd8-7f0441da9256" containerID="f6cfb2aba65d5ff5f3e86cb1bfb489fbafb59488e4ae5a2fde5b45218d975716" exitCode=0
Oct 31 00:56:30.457244 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.457244    4476 generic.go:356] "Generic (PLEG): container finished" podID="f21550fa-496b-4806-abd8-7f0441da9256" containerID="4f094334431c240d11fe029d8b19cdaf3f2c6061c986816323fa100072fe0bb1" exitCode=0
Oct 31 00:56:30.457244 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.457249    4476 generic.go:356] "Generic (PLEG): container finished" podID="f21550fa-496b-4806-abd8-7f0441da9256" containerID="e368fed69a55bc1943e2981ee75c1d43ad2757da847dd3112c779423c6a371cc" exitCode=0
Oct 31 00:56:30.457244 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.457254    4476 generic.go:356] "Generic (PLEG): container finished" podID="f21550fa-496b-4806-abd8-7f0441da9256" containerID="f472e72a2dad76861bbe191902e76df78f2a5ad861c4f72067d20ae87b24112b" exitCode=0
Oct 31 00:56:30.457599 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.457260    4476 generic.go:356] "Generic (PLEG): container finished" podID="f21550fa-496b-4806-abd8-7f0441da9256" containerID="004cf2532b77c7fdb0155215d2b84048f53f7fadd1a3199ddf6eabf122137756" exitCode=0
Oct 31 00:56:30.462934 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.462905    4476 generic.go:356] "Generic (PLEG): container finished" podID="52cc4e07643ef7c5669e5939f9f4241e" containerID="6e8ac3262e8d2f0bee1359580c2a7129de6fdd9d7d1a774a4c5299400700fdc9" exitCode=0
Oct 31 00:56:30.486210 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.486168    4476 manager.go:324] Recovery completed
Oct 31 00:56:30.505313 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.505286    4476 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:56:30.506587 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.506558    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:56:30.506669 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.506597    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:56:30.506669 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.506608    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:56:30.508466 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.508449    4476 cpu_manager.go:226] "Starting" policy="none"
Oct 31 00:56:30.508466 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.508462    4476 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 00:56:30.508601 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.508470    4476 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:56:30.508601 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.508478    4476 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 00:56:30.508601 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.508595    4476 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 00:56:30.508702 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.508606    4476 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 00:56:30.508702 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.508622    4476 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 00:56:30.508702 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.508628    4476 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 00:56:30.508702 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.508633    4476 policy_none.go:50] "Start"
Oct 31 00:56:30.508702 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.508641    4476 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 00:56:30.508702 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.508650    4476 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 00:56:30.508873 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.508771    4476 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 00:56:30.508873 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.508778    4476 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 00:56:30.508873 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.508784    4476 policy_none.go:44] "Start"
Oct 31 00:56:30.509319 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.509303    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:56:30.509372 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.509334    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:56:30.509372 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.509352    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:56:30.511581 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:30.511565    4476 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-174.us-west-1.compute.internal\" not found"
Oct 31 00:56:30.513274 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.513246    4476 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 00:56:30.515934 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.515919    4476 manager.go:341] "Starting Device Plugin manager"
Oct 31 00:56:30.516019 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:30.515949    4476 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 00:56:30.516019 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.515960    4476 server.go:87] "Starting device plugin registration server"
Oct 31 00:56:30.516321 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.516304    4476 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 00:56:30.516420 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.516323    4476 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 00:56:30.516611 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.516592    4476 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 00:56:30.516727 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.516695    4476 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:56:30.516784 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.516737    4476 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 00:56:30.516784 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.516744    4476 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 00:56:30.516863 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.516792    4476 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:56:30.516726918Z","UUID":"e00320d4-2767-47c7-8eea-20c19ff8bf5e","Handler":null,"Name":"","Endpoint":""}
Oct 31 00:56:30.519855 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.518206    4476 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 00:56:30.519855 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.518241    4476 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:56:30.519855 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:30.518666    4476 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 00:56:30.519855 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:30.518699    4476 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-174.us-west-1.compute.internal\" not found"
Oct 31 00:56:30.549980 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.549387    4476 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal"]
Oct 31 00:56:30.550198 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.550088    4476 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:56:30.551337 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.551314    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:56:30.551439 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.551350    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:56:30.551439 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.551367    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:56:30.551708 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.551690    4476 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:56:30.552014 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.551991    4476 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:56:30.553476 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.553449    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:56:30.553683 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.553654    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:56:30.553786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.553689    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:56:30.554098 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.554036    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal" event={"ID":"52cc4e07643ef7c5669e5939f9f4241e","Type":"ContainerStarted","Data":"09dd2b8b41fc4a0154584d33f7a3d43914179db9d812bb3591836b450042ca28"}
Oct 31 00:56:30.554156 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.554125    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal" event={"ID":"52cc4e07643ef7c5669e5939f9f4241e","Type":"ContainerDied","Data":"6e8ac3262e8d2f0bee1359580c2a7129de6fdd9d7d1a774a4c5299400700fdc9"}
Oct 31 00:56:30.554228 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.554154    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal" event={"ID":"52cc4e07643ef7c5669e5939f9f4241e","Type":"ContainerStarted","Data":"c4459dedf25045e99b03c872e7883f37e2c7ddef022171c54d199e6d20e75bab"}
Oct 31 00:56:30.554273 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.554239    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal" event={"ID":"7fc072a2ae8e85ec3464a94d2bac4401","Type":"ContainerStarted","Data":"4e6e6b0b92803e591a81bc405bb426c636208ffce7fedf75efa0220644e87eeb"}
Oct 31 00:56:30.554273 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.554258    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal" event={"ID":"7fc072a2ae8e85ec3464a94d2bac4401","Type":"ContainerStarted","Data":"aaf8df7b834dd3e771643f1bc0143c52b7f25f9bcfb196d2c85069811d3ca94f"}
Oct 31 00:56:30.554394 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.554381    4476 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:56:30.554498 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.553392    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:56:30.554546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.554513    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:56:30.554546 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.554537    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:56:30.554694 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:30.554681    4476 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-174.us-west-1.compute.internal\" not found" node="ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.555329 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.555313    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:56:30.555405 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.555338    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:56:30.555405 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.555348    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:56:30.555490 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:30.555459    4476 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-174.us-west-1.compute.internal\" not found" node="ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.616253 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.611131    4476 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 00:56:30.616253 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.611253    4476 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:56:30.618062 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.617699    4476 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:56:30.619019 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.618993    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:56:30.619159 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.619032    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:56:30.619159 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.619046    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:56:30.619159 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.619095    4476 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.656288 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.656258    4476 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:56:30.660639 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.660594    4476 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:56:30.661730 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.661704    4476 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:56:30.712074 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.712035    4476 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.712258 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.712136    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52cc4e07643ef7c5669e5939f9f4241e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"52cc4e07643ef7c5669e5939f9f4241e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.712258 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.712167    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52cc4e07643ef7c5669e5939f9f4241e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"52cc4e07643ef7c5669e5939f9f4241e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.712391 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.712253    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fc072a2ae8e85ec3464a94d2bac4401-config\") pod \"kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"7fc072a2ae8e85ec3464a94d2bac4401\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.738319 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.738294    4476 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.738473 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.738377    4476 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.739019 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.738985    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeNotReady"
Oct 31 00:56:30.739224 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.739035    4476 setters.go:546] "Node became not ready" node="ip-10-0-139-174.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T00:56:30Z","lastTransitionTime":"2026-10-31T00:56:30Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 31 00:56:30.778501 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.778473    4476 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:56:30.778658 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:30.778529    4476 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.778658 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.778545    4476 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.778807 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.778789    4476 nodeinfomanager.go:471] existing CSINode "ip-10-0-139-174.us-west-1.compute.internal" is owned by different node (oldNodeID="94132dca-c40c-45e0-9b80-e3589c05bc9f", newNodeID=""), cleaning up...
Oct 31 00:56:30.812467 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.812427    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52cc4e07643ef7c5669e5939f9f4241e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"52cc4e07643ef7c5669e5939f9f4241e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.812640 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.812423    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52cc4e07643ef7c5669e5939f9f4241e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"52cc4e07643ef7c5669e5939f9f4241e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.812640 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.812503    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52cc4e07643ef7c5669e5939f9f4241e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"52cc4e07643ef7c5669e5939f9f4241e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.812640 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.812593    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fc072a2ae8e85ec3464a94d2bac4401-config\") pod \"kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"7fc072a2ae8e85ec3464a94d2bac4401\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.812777 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.812652    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fc072a2ae8e85ec3464a94d2bac4401-config\") pod \"kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"7fc072a2ae8e85ec3464a94d2bac4401\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.812777 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.812665    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52cc4e07643ef7c5669e5939f9f4241e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal\" (UID: \"52cc4e07643ef7c5669e5939f9f4241e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:30.837724 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:30.837696    4476 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:56:30.837880 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:30.837739    4476 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:31.017042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.016967    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-174.us-west-1.compute.internal" event="NodeReady"
Oct 31 00:56:31.017210 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.017069    4476 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 00:56:31.398450 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.398415    4476 apiserver.go:52] "Watching apiserver"
Oct 31 00:56:31.486434 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.486398    4476 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:31.503634 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.503601    4476 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 00:56:31.504117 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.504092    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5lczz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-46s2r","openshift-image-registry/node-ca-lthm8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj","openshift-multus/network-metrics-daemon-rlr9b","kube-system/konnectivity-agent-jww65","openshift-insights/insights-runtime-extractor-vqn7z","kube-system/global-pull-secret-syncer-jdmsp","openshift-cluster-node-tuning-operator/tuned-n2x8d","openshift-service-ca/service-ca-5fffc97bdf-9l9rr","openshift-multus/multus-additional-cni-plugins-g2k97","openshift-network-diagnostics/network-check-target-qhvwh","openshift-network-operator/iptables-alerter-djz6l","openshift-dns/node-resolver-gsv6r","openshift-multus/multus-7rmmh","kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal","openshift-ingress-canary/ingress-canary-dljpn","openshift-insights/periodic-gathering-dsx4h-8b9gr"]
Oct 31 00:56:31.510866 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.507619    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:31.516547 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.516497    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-run-netns\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.516699 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.516553    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-etc-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.516699 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.516623    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-ovn\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.516789 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.516717    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-log-socket\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.517096 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.517078    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-ovnkube-config\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.517150 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.517114    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-slash\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.517249 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.517226    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-var-lib-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.517328 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.517280    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.517390 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.517333    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-cni-bin\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.517390 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.517375    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-cni-netd\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.517471 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.517419    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-kubelet\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.517548 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.517465    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-systemd-units\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.517548 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.517511    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-systemd\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.517608 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.517551    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.517608 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.517577    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-node-log\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.517608 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.517594    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-run-ovn-kubernetes\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.518212 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.518194    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:31.518991 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.518972    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.562516 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.562463    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:31.565382 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.565367    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 00:56:31.565609 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.565597    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 00:56:31.565887 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.565867    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 00:56:31.567243 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.567227    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 00:56:31.567408 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.567392    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-ovnkube-config\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.570742 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.570727    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 00:56:31.571511 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.571498    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 00:56:31.571559 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.571516    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lhvrj\""
Oct 31 00:56:31.571559 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.571546    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hnbxm\""
Oct 31 00:56:31.574933 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.574913    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 00:56:31.574933 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.574916    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 00:56:31.575040 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.574945    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ndfs4\""
Oct 31 00:56:31.575233 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.575197    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fksdd\""
Oct 31 00:56:31.601102 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.601071    4476 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:56:31.601276 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:31.601123    4476 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-174.us-west-1.compute.internal"
Oct 31 00:56:31.609374 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.609343    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 00:56:31.609536 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.609469    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 00:56:31.609616 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.609554    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 00:56:31.609616 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.609559    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 00:56:31.609616 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.609596    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 00:56:31.609732 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.609612    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 00:56:31.609732 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.609624    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:56:31.609732 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.609618    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:56:31.611402 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.611389    4476 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 00:56:31.618063 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618046    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-kubernetes\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.618119 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618070    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-systemd\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.618119 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618086    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-var-lib-kubelet\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.618206 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618129    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4311d66-bb16-4a76-a021-527f4c1395f4-serviceca\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:56:31.618241 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618220    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.618333 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618316    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:31.618387 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618354    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-run-netns\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.618450 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618384    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-etc-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.618450 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618407    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-run-netns\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.618450 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618409    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-ovn\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.618450 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618437    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rz6k\" (UniqueName: \"kubernetes.io/projected/27c82039-550c-4fe8-96c7-dfd8990440e4-kube-api-access-7rz6k\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.618620 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618457    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-ovn\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.618620 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618462    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:31.618620 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618483    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-etc-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.618620 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618514    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-config-volume\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:31.618620 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618600    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysctl-conf\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.618854 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618633    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-var-lib-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.618854 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618665    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.618854 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618687    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-cni-bin\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.618854 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618714    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysctl-d\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.618854 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618727    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.618854 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618735    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-device-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.618854 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618743    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-var-lib-openvswitch\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.618854 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618756    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-cni-bin\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.618854 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618799    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.619107 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618925    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-tmp-dir\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:31.619107 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618982    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-systemd-units\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.619107 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.618997    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-tmp-dir\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:31.619107 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619022    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-systemd\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.619107 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619030    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0cade88-4da4-467c-9656-cce6afe1dd62-original-pull-secret\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:31.619107 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619035    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-systemd-units\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.619107 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619046    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-run\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.619107 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619070    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-run-systemd\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.619107 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619077    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/12564994-4d47-4401-bc99-a4755b7ed33f-multus-daemon-config\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.619365 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619125    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wd9l\" (UniqueName: \"kubernetes.io/projected/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-kube-api-access-4wd9l\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:31.619365 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619169    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-run-ovn-kubernetes\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.619365 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619290    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:31.619365 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619294    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-run-ovn-kubernetes\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.619365 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619361    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-system-cni-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.619501 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619378    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.619501 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619395    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e39c2f85-6b3b-4343-bb84-c531243a36e5-cert\") pod \"ingress-canary-dljpn\" (UID: \"e39c2f85-6b3b-4343-bb84-c531243a36e5\") " pod="openshift-ingress-canary/ingress-canary-dljpn"
Oct 31 00:56:31.619501 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619413    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-log-socket\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.619501 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619457    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-lib-modules\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.619501 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619488    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-log-socket\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.619640 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619518    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-host\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.619640 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619550    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-socket-dir-parent\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.619640 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619592    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdcx7\" (UniqueName: \"kubernetes.io/projected/e39c2f85-6b3b-4343-bb84-c531243a36e5-kube-api-access-jdcx7\") pod \"ingress-canary-dljpn\" (UID: \"e39c2f85-6b3b-4343-bb84-c531243a36e5\") " pod="openshift-ingress-canary/ingress-canary-dljpn"
Oct 31 00:56:31.619640 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619608    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.619640 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619623    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-cni-netd\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.619640 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619637    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-env-overrides\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.619817 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619672    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-cni-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.619817 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619697    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-cni-netd\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.619817 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619702    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12564994-4d47-4401-bc99-a4755b7ed33f-cni-binary-copy\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.619817 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619755    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-netns\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.619817 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619810    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-env-overrides\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.620055 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619807    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t522b\" (UniqueName: \"kubernetes.io/projected/e4311d66-bb16-4a76-a021-527f4c1395f4-kube-api-access-t522b\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:56:31.620055 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619850    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.620055 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619887    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-k8s-cni-cncf-io\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.620055 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619914    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-crio-socket\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.620055 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619951    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.620055 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.619987    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4311d66-bb16-4a76-a021-527f4c1395f4-host\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:56:31.620055 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620010    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-registration-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.620055 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620027    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.620284 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620085    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.620284 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620109    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-system-cni-dir\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.620284 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620127    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/506d546e-6187-4c6a-967b-09273e1fa785-archives-path\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:31.620284 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620143    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-ovnkube-script-lib\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.620284 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620212    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/506d546e-6187-4c6a-967b-09273e1fa785-archives-path\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:31.620284 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620262    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-conf-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.620284 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620281    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-hosts-file\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:56:31.620534 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620299    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-service-ca-bundle\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:31.620534 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620316    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrm7b\" (UniqueName: \"kubernetes.io/projected/506d546e-6187-4c6a-967b-09273e1fa785-kube-api-access-nrm7b\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:31.620534 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620333    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-542pt\" (UniqueName: \"kubernetes.io/projected/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-kube-api-access-542pt\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:31.620534 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620373    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-slash\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.620534 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620376    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68af8611-a641-4663-b13b-8f015107e82f-ovnkube-script-lib\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.620534 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620417    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-slash\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.620534 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620454    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-tuned\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.620534 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620532    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-tuned\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.620880 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620511    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9mcc\" (UniqueName: \"kubernetes.io/projected/9bd7a453-d0b1-4db7-8dfa-247a72467481-kube-api-access-q9mcc\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.620880 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620602    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7vl\" (UniqueName: \"kubernetes.io/projected/ac61d127-75ba-4f73-bcc6-7d69bf320a55-kube-api-access-vr7vl\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:56:31.620880 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620708    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-tmp-dir\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:56:31.620880 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620783    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47hxq\" (UniqueName: \"kubernetes.io/projected/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-api-access-47hxq\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.620880 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620831    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-tmp-dir\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:56:31.620880 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620878    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-kubelet\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.621100 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620917    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bd7a453-d0b1-4db7-8dfa-247a72467481-tmp\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.621100 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620941    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-host-kubelet\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.621100 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620943    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-cni-bin\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.621100 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620967    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-multus-certs\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.621100 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620990    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bd7a453-d0b1-4db7-8dfa-247a72467481-tmp\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.621100 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.620990    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac61d127-75ba-4f73-bcc6-7d69bf320a55-host-slash\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:56:31.621100 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621030    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdzc\" (UniqueName: \"kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc\") pod \"network-check-target-qhvwh\" (UID: \"74673206-8b68-4414-8ac0-a63577cea44a\") " pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:31.621100 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621060    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfxdb\" (UniqueName: \"kubernetes.io/projected/68af8611-a641-4663-b13b-8f015107e82f-kube-api-access-kfxdb\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.621375 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621108    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-sys\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.621375 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621160    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-etc-kubernetes\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.621375 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621260    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b893def-2fc6-4350-a01f-6ee4969c48bd-agent-certs\") pod \"konnectivity-agent-jww65\" (UID: \"7b893def-2fc6-4350-a01f-6ee4969c48bd\") " pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:56:31.621375 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621311    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b893def-2fc6-4350-a01f-6ee4969c48bd-konnectivity-ca\") pod \"konnectivity-agent-jww65\" (UID: \"7b893def-2fc6-4350-a01f-6ee4969c48bd\") " pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:56:31.621375 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621347    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0cade88-4da4-467c-9656-cce6afe1dd62-kubelet-config\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:31.621564 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621390    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0cade88-4da4-467c-9656-cce6afe1dd62-dbus\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:31.621564 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621439    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cphjb\" (UniqueName: \"kubernetes.io/projected/f21550fa-496b-4806-abd8-7f0441da9256-kube-api-access-cphjb\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.621564 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621471    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-cni-multus\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.621564 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621495    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.621706 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621568    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b893def-2fc6-4350-a01f-6ee4969c48bd-konnectivity-ca\") pod \"konnectivity-agent-jww65\" (UID: \"7b893def-2fc6-4350-a01f-6ee4969c48bd\") " pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:56:31.621706 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621618    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shw5k\" (UniqueName: \"kubernetes.io/projected/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-kube-api-access-shw5k\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:31.621706 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621692    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-signing-key\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:31.621803 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621711    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-signing-cabundle\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:31.621856 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621841    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68af8611-a641-4663-b13b-8f015107e82f-ovn-node-metrics-cert\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.621895 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621876    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-cnibin\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.621895 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621893    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-kubelet\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.621985 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621908    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-hostroot\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.621985 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621942    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86lw\" (UniqueName: \"kubernetes.io/projected/12564994-4d47-4401-bc99-a4755b7ed33f-kube-api-access-h86lw\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.621985 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.621970    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-socket-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.622122 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.622016    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-data-volume\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.622122 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.622105    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-cnibin\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.622224 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.622126    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68af8611-a641-4663-b13b-8f015107e82f-ovn-node-metrics-cert\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.622224 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.622143    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-modprobe-d\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.622224 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.622162    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysconfig\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.622224 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.622148    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-data-volume\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.622224 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.622210    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac61d127-75ba-4f73-bcc6-7d69bf320a55-iptables-alerter-script\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:56:31.622407 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.622230    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-sys-fs\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.622407 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.622250    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vwtr\" (UniqueName: \"kubernetes.io/projected/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-kube-api-access-2vwtr\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:56:31.622407 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.622264    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.622407 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.622279    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-trusted-ca-bundle\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:31.622407 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.622336    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-node-log\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.622407 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.622395    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68af8611-a641-4663-b13b-8f015107e82f-node-log\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.622585 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.622415    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac61d127-75ba-4f73-bcc6-7d69bf320a55-iptables-alerter-script\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:56:31.638878 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.638853    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 00:56:31.639983 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.639959    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 00:56:31.640309 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.640282    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 00:56:31.640415 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.640326    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vxv7v\""
Oct 31 00:56:31.640415 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.640342    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 00:56:31.640415 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.640355    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 00:56:31.640572 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.640465    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 00:56:31.640572 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.640473    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 00:56:31.640572 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.640493    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 00:56:31.640572 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.640518    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 00:56:31.640572 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.640532    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-98n4j\""
Oct 31 00:56:31.640572 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.640536    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 00:56:31.641652 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.641637    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b893def-2fc6-4350-a01f-6ee4969c48bd-agent-certs\") pod \"konnectivity-agent-jww65\" (UID: \"7b893def-2fc6-4350-a01f-6ee4969c48bd\") " pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:56:31.649099 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.649046    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4311d66-bb16-4a76-a021-527f4c1395f4-serviceca\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:56:31.655060 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.655043    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6sjwh\""
Oct 31 00:56:31.656395 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.656374    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 00:56:31.656395 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.656383    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 00:56:31.656518 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.656452    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g8n5b\""
Oct 31 00:56:31.656518 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.656463    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 00:56:31.656578 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.656519    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wfh75\""
Oct 31 00:56:31.657059 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.657046    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 00:56:31.657891 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.657878    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 00:56:31.658477 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.658461    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 00:56:31.658477 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.658470    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 00:56:31.658570 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.658501    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 00:56:31.658570 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.658510    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 00:56:31.659822 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.659801    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/12564994-4d47-4401-bc99-a4755b7ed33f-multus-daemon-config\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.659986 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.659970    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12564994-4d47-4401-bc99-a4755b7ed33f-cni-binary-copy\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.661076 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.661062    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xr8df\""
Oct 31 00:56:31.661128 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.661072    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-service-ca-bundle\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:31.662141 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.662123    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-signing-cabundle\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:31.662221 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.662202    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.666231 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.666211    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gjj6f\""
Oct 31 00:56:31.666727 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.666709    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 00:56:31.666808 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.666760    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 00:56:31.666808 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.666771    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 00:56:31.666913 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.666902    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 00:56:31.667728 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.667714    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 00:56:31.667792 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.667714    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 00:56:31.668036 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.668012    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 00:56:31.668218 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.668117    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rf2ln\""
Oct 31 00:56:31.668218 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.668141    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 00:56:31.668218 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.668166    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nktwg\""
Oct 31 00:56:31.668349 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.668283    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gr6qq\""
Oct 31 00:56:31.668349 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.668328    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 00:56:31.668417 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.668358    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 00:56:31.668417 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.668399    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sk5g6\""
Oct 31 00:56:31.668417 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.668399    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m5b2l\""
Oct 31 00:56:31.668520 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.668458    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dj4kh\""
Oct 31 00:56:31.668561 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.668544    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.669421 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.669404    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-config-volume\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:31.670569 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.670555    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e39c2f85-6b3b-4343-bb84-c531243a36e5-cert\") pod \"ingress-canary-dljpn\" (UID: \"e39c2f85-6b3b-4343-bb84-c531243a36e5\") " pod="openshift-ingress-canary/ingress-canary-dljpn"
Oct 31 00:56:31.672190 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.672166    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-signing-key\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:31.673332 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.673314    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f21550fa-496b-4806-abd8-7f0441da9256-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.717060 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.717031    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 00:56:31.723267 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723229    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-kubelet\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.723399 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723275    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-hostroot\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.723399 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723310    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-trusted-ca-bundle\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:31.723399 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723331    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-socket-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.723399 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723336    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-kubelet\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.723399 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723377    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-cnibin\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.723648 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723392    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-hostroot\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.723648 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723434    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-socket-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.723648 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723449    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-cnibin\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.723648 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723454    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-modprobe-d\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.723648 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723502    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysconfig\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.723648 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723552    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-modprobe-d\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.723648 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723596    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-sys-fs\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.723648 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723632    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-kubernetes\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723655    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysconfig\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723666    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-systemd\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723681    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-sys-fs\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723698    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-var-lib-kubelet\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723720    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-systemd\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723746    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723757    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-var-lib-kubelet\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723718    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-kubernetes\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723777    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysctl-conf\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723801    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysctl-d\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723821    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-device-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723842    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723864    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-run\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723901    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysctl-conf\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723936    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-device-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.723965    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-etc-sysctl-d\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724006    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:31.724042 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724036    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-run\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724036    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-system-cni-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724097    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:31.724140    4476 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724153    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-system-cni-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724204    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724140    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-lib-modules\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724275    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-lib-modules\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:31.724281    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs podName:9d67aaf2-9211-4f6c-84d8-99aae85c28eb nodeName:}" failed. No retries permitted until 2026-10-31 00:56:32.224258703 +0000 UTC m=+1.914141833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs") pod "network-metrics-daemon-rlr9b" (UID: "9d67aaf2-9211-4f6c-84d8-99aae85c28eb") : secret "metrics-daemon-secret" not found
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724280    4476 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724321    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-host\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724351    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-socket-dir-parent\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724381    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724394    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-host\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724404    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-cni-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724498    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-netns\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724492    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724507    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-cni-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724505    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-socket-dir-parent\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724532    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-k8s-cni-cncf-io\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724568    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-crio-socket\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724574    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-k8s-cni-cncf-io\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724614    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-netns\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724617    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4311d66-bb16-4a76-a021-527f4c1395f4-host\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724635    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4311d66-bb16-4a76-a021-527f4c1395f4-host\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724652    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-registration-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724685    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724694    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-registration-dir\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724723    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724728    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-crio-socket\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724739    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27c82039-550c-4fe8-96c7-dfd8990440e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724759    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-system-cni-dir\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724795    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-conf-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.724834 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724830    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-hosts-file\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724862    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-multus-conf-dir\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724881    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f21550fa-496b-4806-abd8-7f0441da9256-system-cni-dir\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724922    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-hosts-file\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724877    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-cni-bin\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724946    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-cni-bin\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724959    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-multus-certs\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724979    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac61d127-75ba-4f73-bcc6-7d69bf320a55-host-slash\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.724993    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-sys\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.725007    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-etc-kubernetes\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.725007    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-run-multus-certs\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.725033    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0cade88-4da4-467c-9656-cce6afe1dd62-kubelet-config\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.725050    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac61d127-75ba-4f73-bcc6-7d69bf320a55-host-slash\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.725075    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0cade88-4da4-467c-9656-cce6afe1dd62-dbus\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.725106    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-etc-kubernetes\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.725114    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-cni-multus\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.725155    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0cade88-4da4-467c-9656-cce6afe1dd62-dbus\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.725169    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-cnibin\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.725203    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0cade88-4da4-467c-9656-cce6afe1dd62-kubelet-config\") pod \"global-pull-secret-syncer-jdmsp\" (UID: \"e0cade88-4da4-467c-9656-cce6afe1dd62\") " pod="kube-system/global-pull-secret-syncer-jdmsp"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.725088    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bd7a453-d0b1-4db7-8dfa-247a72467481-sys\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.725279    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.725321    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-cnibin\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.726311 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.725321    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12564994-4d47-4401-bc99-a4755b7ed33f-host-var-lib-cni-multus\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.727259 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.727161    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-metrics-tls\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:31.727312 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.727289    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.729030 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.729002    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t522b\" (UniqueName: \"kubernetes.io/projected/e4311d66-bb16-4a76-a021-527f4c1395f4-kube-api-access-t522b\") pod \"node-ca-lthm8\" (UID: \"e4311d66-bb16-4a76-a021-527f4c1395f4\") " pod="openshift-image-registry/node-ca-lthm8"
Oct 31 00:56:31.729734 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.729714    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr7vl\" (UniqueName: \"kubernetes.io/projected/ac61d127-75ba-4f73-bcc6-7d69bf320a55-kube-api-access-vr7vl\") pod \"iptables-alerter-djz6l\" (UID: \"ac61d127-75ba-4f73-bcc6-7d69bf320a55\") " pod="openshift-network-operator/iptables-alerter-djz6l"
Oct 31 00:56:31.734111 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.734092    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47hxq\" (UniqueName: \"kubernetes.io/projected/7bf35a3c-66f9-4ccf-b340-50a072ffa61f-kube-api-access-47hxq\") pod \"insights-runtime-extractor-vqn7z\" (UID: \"7bf35a3c-66f9-4ccf-b340-50a072ffa61f\") " pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.736645 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.736001    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86lw\" (UniqueName: \"kubernetes.io/projected/12564994-4d47-4401-bc99-a4755b7ed33f-kube-api-access-h86lw\") pod \"multus-7rmmh\" (UID: \"12564994-4d47-4401-bc99-a4755b7ed33f\") " pod="openshift-multus/multus-7rmmh"
Oct 31 00:56:31.737063 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.736966    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdcx7\" (UniqueName: \"kubernetes.io/projected/e39c2f85-6b3b-4343-bb84-c531243a36e5-kube-api-access-jdcx7\") pod \"ingress-canary-dljpn\" (UID: \"e39c2f85-6b3b-4343-bb84-c531243a36e5\") " pod="openshift-ingress-canary/ingress-canary-dljpn"
Oct 31 00:56:31.737519 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.737447    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shw5k\" (UniqueName: \"kubernetes.io/projected/f90cbecc-43bb-41a8-b743-4cdcd12dce2f-kube-api-access-shw5k\") pod \"dns-default-5lczz\" (UID: \"f90cbecc-43bb-41a8-b743-4cdcd12dce2f\") " pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:31.738961 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.737716    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wd9l\" (UniqueName: \"kubernetes.io/projected/36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c-kube-api-access-4wd9l\") pod \"service-ca-5fffc97bdf-9l9rr\" (UID: \"36ab3f3f-3f1d-4843-87ce-6b5a4e81ad1c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9l9rr"
Oct 31 00:56:31.738961 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.737834    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrm7b\" (UniqueName: \"kubernetes.io/projected/506d546e-6187-4c6a-967b-09273e1fa785-kube-api-access-nrm7b\") pod \"periodic-gathering-dsx4h-8b9gr\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") " pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:56:31.738961 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.737979    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cphjb\" (UniqueName: \"kubernetes.io/projected/f21550fa-496b-4806-abd8-7f0441da9256-kube-api-access-cphjb\") pod \"multus-additional-cni-plugins-g2k97\" (UID: \"f21550fa-496b-4806-abd8-7f0441da9256\") " pod="openshift-multus/multus-additional-cni-plugins-g2k97"
Oct 31 00:56:31.738961 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.738272    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vwtr\" (UniqueName: \"kubernetes.io/projected/7ae98732-67f4-420b-9e5b-e4e311ac9f9c-kube-api-access-2vwtr\") pod \"node-resolver-gsv6r\" (UID: \"7ae98732-67f4-420b-9e5b-e4e311ac9f9c\") " pod="openshift-dns/node-resolver-gsv6r"
Oct 31 00:56:31.738961 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.738473    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pdzc\" (UniqueName: \"kubernetes.io/projected/74673206-8b68-4414-8ac0-a63577cea44a-kube-api-access-5pdzc\") pod \"network-check-target-qhvwh\" (UID: \"74673206-8b68-4414-8ac0-a63577cea44a\") " pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:31.739964 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.739922    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-542pt\" (UniqueName: \"kubernetes.io/projected/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-kube-api-access-542pt\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:31.741097 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.741075    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rz6k\" (UniqueName: \"kubernetes.io/projected/27c82039-550c-4fe8-96c7-dfd8990440e4-kube-api-access-7rz6k\") pod \"aws-ebs-csi-driver-node-7wjxj\" (UID: \"27c82039-550c-4fe8-96c7-dfd8990440e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7wjxj"
Oct 31 00:56:31.741198 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.741157    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfxdb\" (UniqueName: \"kubernetes.io/projected/68af8611-a641-4663-b13b-8f015107e82f-kube-api-access-kfxdb\") pod \"ovnkube-node-46s2r\" (UID: \"68af8611-a641-4663-b13b-8f015107e82f\") " pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:31.742766 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.742748    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9mcc\" (UniqueName: \"kubernetes.io/projected/9bd7a453-d0b1-4db7-8dfa-247a72467481-kube-api-access-q9mcc\") pod \"tuned-n2x8d\" (UID: \"9bd7a453-d0b1-4db7-8dfa-247a72467481\") " pod="openshift-cluster-node-tuning-operator/tuned-n2x8d"
Oct 31 00:56:31.777081 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.777055    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lthm8_e4311d66-bb16-4a76-a021-527f4c1395f4/node-ca/0.log"
Oct 31 00:56:31.819139 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.819116    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:31.820245 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.820224    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vqn7z"
Oct 31 00:56:31.858280 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:31.858246    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:32.132412 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:32.132377    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:32.223908 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:32.223820    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5lczz"]
Oct 31 00:56:32.229657 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:32.229634    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:32.229787 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:32.229757    4476 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:56:32.229844 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:32.229826    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs podName:9d67aaf2-9211-4f6c-84d8-99aae85c28eb nodeName:}" failed. No retries permitted until 2026-10-31 00:56:33.229807425 +0000 UTC m=+2.919690547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs") pod "network-metrics-daemon-rlr9b" (UID: "9d67aaf2-9211-4f6c-84d8-99aae85c28eb") : secret "metrics-daemon-secret" not found
Oct 31 00:56:32.260922 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:32.260882    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90cbecc_43bb_41a8_b743_4cdcd12dce2f.slice/crio-bec27159d6029b7cc345bfcf91df87ffa49129d6d95836e980d56c8e43f24940 WatchSource:0}: Error finding container bec27159d6029b7cc345bfcf91df87ffa49129d6d95836e980d56c8e43f24940: Status 404 returned error can't find the container with id bec27159d6029b7cc345bfcf91df87ffa49129d6d95836e980d56c8e43f24940
Oct 31 00:56:32.262932 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:32.262918    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:56:32.274043 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:32.274023    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vqn7z"]
Oct 31 00:56:32.276666 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:32.276636    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf35a3c_66f9_4ccf_b340_50a072ffa61f.slice/crio-dea916682a50ca5d450da1108f78c68f612940f7afb8a2eaf8ca97507d94e26b WatchSource:0}: Error finding container dea916682a50ca5d450da1108f78c68f612940f7afb8a2eaf8ca97507d94e26b: Status 404 returned error can't find the container with id dea916682a50ca5d450da1108f78c68f612940f7afb8a2eaf8ca97507d94e26b
Oct 31 00:56:32.489843 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:32.489759    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5lczz" event={"ID":"f90cbecc-43bb-41a8-b743-4cdcd12dce2f","Type":"ContainerStarted","Data":"bec27159d6029b7cc345bfcf91df87ffa49129d6d95836e980d56c8e43f24940"}
Oct 31 00:56:32.491397 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:32.491373    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vqn7z" event={"ID":"7bf35a3c-66f9-4ccf-b340-50a072ffa61f","Type":"ContainerStarted","Data":"099dafa4118f1aefd1309d34ccf008beb541d30bf51c95bf71dbb1428b47897c"}
Oct 31 00:56:32.491521 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:32.491423    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vqn7z" event={"ID":"7bf35a3c-66f9-4ccf-b340-50a072ffa61f","Type":"ContainerStarted","Data":"dea916682a50ca5d450da1108f78c68f612940f7afb8a2eaf8ca97507d94e26b"}
Oct 31 00:56:32.640434 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:32.640400    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:32.777732 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:32.777713    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dljpn_e39c2f85-6b3b-4343-bb84-c531243a36e5/serve-healthcheck-canary/0.log"
Oct 31 00:56:33.055801 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:33.055718    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:33.237739 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:33.237704    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:33.237907 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:33.237842    4476 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:56:33.237967 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:33.237925    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs podName:9d67aaf2-9211-4f6c-84d8-99aae85c28eb nodeName:}" failed. No retries permitted until 2026-10-31 00:56:35.237907155 +0000 UTC m=+4.927790277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs") pod "network-metrics-daemon-rlr9b" (UID: "9d67aaf2-9211-4f6c-84d8-99aae85c28eb") : secret "metrics-daemon-secret" not found
Oct 31 00:56:33.289799 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:33.289743    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:33.498529 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:33.498447    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:33.713750 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:33.713708    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:33.918028 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:33.917999    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:33.995915 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:33.995893    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:34.083986 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:34.083962    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jww65"
Oct 31 00:56:34.500076 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:34.500040    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vqn7z" event={"ID":"7bf35a3c-66f9-4ccf-b340-50a072ffa61f","Type":"ContainerStarted","Data":"ea9da8f245755380f926dfd432680af7ce259ee65a69031fa4c8181948fdb75d"}
Oct 31 00:56:34.638131 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:34.638109    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:35.038453 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:35.038420    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:35.238149 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:35.238111    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:35.254084 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:35.254055    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:35.254283 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:35.254169    4476 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:56:35.254283 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:35.254255    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs podName:9d67aaf2-9211-4f6c-84d8-99aae85c28eb nodeName:}" failed. No retries permitted until 2026-10-31 00:56:39.254236106 +0000 UTC m=+8.944119226 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs") pod "network-metrics-daemon-rlr9b" (UID: "9d67aaf2-9211-4f6c-84d8-99aae85c28eb") : secret "metrics-daemon-secret" not found
Oct 31 00:56:35.447396 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:35.447319    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:35.505928 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:35.505895    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5lczz" event={"ID":"f90cbecc-43bb-41a8-b743-4cdcd12dce2f","Type":"ContainerStarted","Data":"55262df56665da902609bd7d9b29e1bd139fc73b0aaeddec67afc2b9d24a737a"}
Oct 31 00:56:35.506431 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:35.505952    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5lczz" event={"ID":"f90cbecc-43bb-41a8-b743-4cdcd12dce2f","Type":"ContainerStarted","Data":"cc9c214683881c24fe6434cac2c56c470680db0e17ec8603a5fe830f474ac795"}
Oct 31 00:56:35.506431 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:35.506043    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:35.827876 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:35.827832    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:36.513552 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:36.513511    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vqn7z" event={"ID":"7bf35a3c-66f9-4ccf-b340-50a072ffa61f","Type":"ContainerStarted","Data":"e2d8c10e3e37355c8d04fbeedd4159018983c28ecb1e632b7153bab00d060699"}
Oct 31 00:56:36.642000 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:36.641965    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:37.085202 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:37.085161    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:37.375446 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:37.375371    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:37.579358 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:37.579323    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:37.856825 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:37.856791    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:38.562375 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:38.562348    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:38.812366 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:38.812330    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:39.049427 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:39.049389    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:39.264809 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:39.264777    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:39.282786 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:39.282758    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:39.282898 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:39.282876    4476 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:56:39.282957 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:39.282949    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs podName:9d67aaf2-9211-4f6c-84d8-99aae85c28eb nodeName:}" failed. No retries permitted until 2026-10-31 00:56:47.282935672 +0000 UTC m=+16.972818777 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs") pod "network-metrics-daemon-rlr9b" (UID: "9d67aaf2-9211-4f6c-84d8-99aae85c28eb") : secret "metrics-daemon-secret" not found
Oct 31 00:56:39.478684 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:39.478604    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:39.735165 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:39.735094    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:39.992386 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:39.992306    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:40.226879 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:40.226849    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:40.414187 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:40.414143    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:40.414381 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:40.414369    4476 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:56:40.435062 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:40.435036    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:56:40.569010 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:40.568981    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:40.787347 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:40.787312    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:40.996218 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:40.996161    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:41.225962 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:41.225879    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:41.445353 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:41.445321    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:41.899623 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:41.899594    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:42.107069 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:42.107039    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:43.039333 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:43.039305    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:43.239771 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:43.239738    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:45.516267 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:45.516240    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5lczz"
Oct 31 00:56:47.335954 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:47.335921    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:47.338381 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:47.338357    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d67aaf2-9211-4f6c-84d8-99aae85c28eb-metrics-certs\") pod \"network-metrics-daemon-rlr9b\" (UID: \"9d67aaf2-9211-4f6c-84d8-99aae85c28eb\") " pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:47.412165 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:47.412131    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlr9b"
Oct 31 00:56:47.788301 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:47.788266    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rlr9b"]
Oct 31 00:56:47.791893 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:47.791862    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d67aaf2_9211_4f6c_84d8_99aae85c28eb.slice/crio-d534ec963da044f6ed5b0c1c6eca902c0e455523b25d59e25d3692b92ec09753 WatchSource:0}: Error finding container d534ec963da044f6ed5b0c1c6eca902c0e455523b25d59e25d3692b92ec09753: Status 404 returned error can't find the container with id d534ec963da044f6ed5b0c1c6eca902c0e455523b25d59e25d3692b92ec09753
Oct 31 00:56:48.432450 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.432418    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-78s6n"]
Oct 31 00:56:48.434839 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.434818    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-78s6n"
Oct 31 00:56:48.452258 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.452226    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tw9gh"]
Oct 31 00:56:48.454011 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.453988    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tw9gh"
Oct 31 00:56:48.481669 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.481640    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xqkgj"]
Oct 31 00:56:48.483335 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.483319    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xqkgj"
Oct 31 00:56:48.501268 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.501246    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 00:56:48.501268 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.501270    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 00:56:48.501428 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.501288    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-jfdjz\""
Oct 31 00:56:48.513652 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.513627    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 00:56:48.530607 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.530585    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 00:56:48.541062 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.541035    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64949dc4d4-2tx9z"]
Oct 31 00:56:48.542955 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.542932    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tw9gh"]
Oct 31 00:56:48.543095 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.543023    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.549105 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.549074    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/75228856-5f8f-4d69-9c35-1012563b9ed1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xqkgj\" (UID: \"75228856-5f8f-4d69-9c35-1012563b9ed1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xqkgj"
Oct 31 00:56:48.549233 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.549135    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvn6m\" (UniqueName: \"kubernetes.io/projected/70365afb-44c6-4aa3-8512-0f01c05ead6e-kube-api-access-jvn6m\") pod \"downloads-6dd789c8b7-78s6n\" (UID: \"70365afb-44c6-4aa3-8512-0f01c05ead6e\") " pod="openshift-console/downloads-6dd789c8b7-78s6n"
Oct 31 00:56:48.549233 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.549160    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h85dc\" (UniqueName: \"kubernetes.io/projected/83e3e872-a804-480e-a267-c59f6ab82512-kube-api-access-h85dc\") pod \"migrator-66df5d496b-tw9gh\" (UID: \"83e3e872-a804-480e-a267-c59f6ab82512\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tw9gh"
Oct 31 00:56:48.549233 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.549224    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70365afb-44c6-4aa3-8512-0f01c05ead6e-tmp\") pod \"downloads-6dd789c8b7-78s6n\" (UID: \"70365afb-44c6-4aa3-8512-0f01c05ead6e\") " pod="openshift-console/downloads-6dd789c8b7-78s6n"
Oct 31 00:56:48.551863 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.551841    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rlr9b" event={"ID":"9d67aaf2-9211-4f6c-84d8-99aae85c28eb","Type":"ContainerStarted","Data":"d534ec963da044f6ed5b0c1c6eca902c0e455523b25d59e25d3692b92ec09753"}
Oct 31 00:56:48.555772 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.555748    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-78s6n"]
Oct 31 00:56:48.555867 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.555801    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xqkgj"]
Oct 31 00:56:48.597951 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.597860    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 00:56:48.598241 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.597954    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-j6xpj\""
Oct 31 00:56:48.598241 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.598143    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-z7n5g\""
Oct 31 00:56:48.650028 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.649997    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 00:56:48.650782 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.650748    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjnfr\" (UniqueName: \"kubernetes.io/projected/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-kube-api-access-sjnfr\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.650905 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.650816    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/75228856-5f8f-4d69-9c35-1012563b9ed1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xqkgj\" (UID: \"75228856-5f8f-4d69-9c35-1012563b9ed1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xqkgj"
Oct 31 00:56:48.650905 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.650848    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-trusted-ca\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.650905 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.650889    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-ca-trust-extracted\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.651064 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.650921    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70365afb-44c6-4aa3-8512-0f01c05ead6e-tmp\") pod \"downloads-6dd789c8b7-78s6n\" (UID: \"70365afb-44c6-4aa3-8512-0f01c05ead6e\") " pod="openshift-console/downloads-6dd789c8b7-78s6n"
Oct 31 00:56:48.651064 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.650951    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-bound-sa-token\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.651064 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.651004    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-image-registry-private-configuration\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.651064 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.651043    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-registry-certificates\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.651273 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.651069    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-installation-pull-secrets\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.651273 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.651095    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-registry-tls\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.651273 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.651124    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvn6m\" (UniqueName: \"kubernetes.io/projected/70365afb-44c6-4aa3-8512-0f01c05ead6e-kube-api-access-jvn6m\") pod \"downloads-6dd789c8b7-78s6n\" (UID: \"70365afb-44c6-4aa3-8512-0f01c05ead6e\") " pod="openshift-console/downloads-6dd789c8b7-78s6n"
Oct 31 00:56:48.651273 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.651153    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h85dc\" (UniqueName: \"kubernetes.io/projected/83e3e872-a804-480e-a267-c59f6ab82512-kube-api-access-h85dc\") pod \"migrator-66df5d496b-tw9gh\" (UID: \"83e3e872-a804-480e-a267-c59f6ab82512\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tw9gh"
Oct 31 00:56:48.653711 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.652711    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70365afb-44c6-4aa3-8512-0f01c05ead6e-tmp\") pod \"downloads-6dd789c8b7-78s6n\" (UID: \"70365afb-44c6-4aa3-8512-0f01c05ead6e\") " pod="openshift-console/downloads-6dd789c8b7-78s6n"
Oct 31 00:56:48.653711 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.653580    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64949dc4d4-2tx9z"]
Oct 31 00:56:48.656401 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.656375    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/75228856-5f8f-4d69-9c35-1012563b9ed1-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xqkgj\" (UID: \"75228856-5f8f-4d69-9c35-1012563b9ed1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xqkgj"
Oct 31 00:56:48.698500 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.698409    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 00:56:48.698670 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.698506    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 00:56:48.698670 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.698600    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 00:56:48.698670 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.698643    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pmlvd\""
Oct 31 00:56:48.710016 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:48.709989    4476 projected.go:291] Couldn't get configMap openshift-kube-storage-version-migrator/openshift-service-ca.crt: configmap "openshift-service-ca.crt" not found
Oct 31 00:56:48.710016 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:48.710014    4476 projected.go:196] Error preparing data for projected volume kube-api-access-h85dc for pod openshift-kube-storage-version-migrator/migrator-66df5d496b-tw9gh: configmap "openshift-service-ca.crt" not found
Oct 31 00:56:48.710227 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:48.710093    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83e3e872-a804-480e-a267-c59f6ab82512-kube-api-access-h85dc podName:83e3e872-a804-480e-a267-c59f6ab82512 nodeName:}" failed. No retries permitted until 2026-10-31 00:56:49.210072192 +0000 UTC m=+18.899955311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h85dc" (UniqueName: "kubernetes.io/projected/83e3e872-a804-480e-a267-c59f6ab82512-kube-api-access-h85dc") pod "migrator-66df5d496b-tw9gh" (UID: "83e3e872-a804-480e-a267-c59f6ab82512") : configmap "openshift-service-ca.crt" not found
Oct 31 00:56:48.711548 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.711529    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvn6m\" (UniqueName: \"kubernetes.io/projected/70365afb-44c6-4aa3-8512-0f01c05ead6e-kube-api-access-jvn6m\") pod \"downloads-6dd789c8b7-78s6n\" (UID: \"70365afb-44c6-4aa3-8512-0f01c05ead6e\") " pod="openshift-console/downloads-6dd789c8b7-78s6n"
Oct 31 00:56:48.745354 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.745325    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-78s6n"
Oct 31 00:56:48.752409 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.752379    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-ca-trust-extracted\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.752552 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.752423    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-bound-sa-token\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.752552 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.752465    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-image-registry-private-configuration\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.752552 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.752502    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-registry-certificates\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.752552 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.752528    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-installation-pull-secrets\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.752761 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.752553    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-registry-tls\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.752761 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.752607    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjnfr\" (UniqueName: \"kubernetes.io/projected/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-kube-api-access-sjnfr\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.752761 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.752645    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-trusted-ca\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.752911 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.752794    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-ca-trust-extracted\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.753433 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.753412    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-registry-certificates\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.753820 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.753796    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-trusted-ca\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.756090 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.756045    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-installation-pull-secrets\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.756509 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.756344    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-image-registry-private-configuration\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.756509 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.756487    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-registry-tls\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.792736 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.792431    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xqkgj"
Oct 31 00:56:48.796400 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.796321    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:48.811321 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.811275    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-bound-sa-token\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.813380 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.813273    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjnfr\" (UniqueName: \"kubernetes.io/projected/634bbaef-6cd6-4686-8aed-f1797ef6c5bc-kube-api-access-sjnfr\") pod \"image-registry-64949dc4d4-2tx9z\" (UID: \"634bbaef-6cd6-4686-8aed-f1797ef6c5bc\") " pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.856021 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.855391    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:48.985530 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:48.985443    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:49.128121 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.128083    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-78s6n"]
Oct 31 00:56:49.131552 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:49.131520    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70365afb_44c6_4aa3_8512_0f01c05ead6e.slice/crio-471a646149a8b7e24c48e7dfda9c82cef891b14bedf264401c632b14e2c2af9a WatchSource:0}: Error finding container 471a646149a8b7e24c48e7dfda9c82cef891b14bedf264401c632b14e2c2af9a: Status 404 returned error can't find the container with id 471a646149a8b7e24c48e7dfda9c82cef891b14bedf264401c632b14e2c2af9a
Oct 31 00:56:49.187831 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.187786    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:49.217635 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.217603    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xqkgj"]
Oct 31 00:56:49.221453 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:49.221425    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75228856_5f8f_4d69_9c35_1012563b9ed1.slice/crio-d6ec5144bfe61c14b5ddd013178ebb414b4a06914e2b5d15ce268a9e7610feb1 WatchSource:0}: Error finding container d6ec5144bfe61c14b5ddd013178ebb414b4a06914e2b5d15ce268a9e7610feb1: Status 404 returned error can't find the container with id d6ec5144bfe61c14b5ddd013178ebb414b4a06914e2b5d15ce268a9e7610feb1
Oct 31 00:56:49.249208 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.249118    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64949dc4d4-2tx9z"]
Oct 31 00:56:49.254771 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:49.254744    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634bbaef_6cd6_4686_8aed_f1797ef6c5bc.slice/crio-92f57f7bfebfc049ae0415b9c0a54b7193f78f9505f5645580d83538d76d15bc WatchSource:0}: Error finding container 92f57f7bfebfc049ae0415b9c0a54b7193f78f9505f5645580d83538d76d15bc: Status 404 returned error can't find the container with id 92f57f7bfebfc049ae0415b9c0a54b7193f78f9505f5645580d83538d76d15bc
Oct 31 00:56:49.259618 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.259591    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h85dc\" (UniqueName: \"kubernetes.io/projected/83e3e872-a804-480e-a267-c59f6ab82512-kube-api-access-h85dc\") pod \"migrator-66df5d496b-tw9gh\" (UID: \"83e3e872-a804-480e-a267-c59f6ab82512\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tw9gh"
Oct 31 00:56:49.262365 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.262338    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h85dc\" (UniqueName: \"kubernetes.io/projected/83e3e872-a804-480e-a267-c59f6ab82512-kube-api-access-h85dc\") pod \"migrator-66df5d496b-tw9gh\" (UID: \"83e3e872-a804-480e-a267-c59f6ab82512\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tw9gh"
Oct 31 00:56:49.363218 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.363193    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tw9gh"
Oct 31 00:56:49.460768 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.460734    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:49.557599 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.557555    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z" event={"ID":"634bbaef-6cd6-4686-8aed-f1797ef6c5bc","Type":"ContainerStarted","Data":"fc7c32642393a9938ff488c22cee97fa2bee010c489c8ce069006708ec6744dc"}
Oct 31 00:56:49.557780 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.557635    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:56:49.557780 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.557661    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z" event={"ID":"634bbaef-6cd6-4686-8aed-f1797ef6c5bc","Type":"ContainerStarted","Data":"92f57f7bfebfc049ae0415b9c0a54b7193f78f9505f5645580d83538d76d15bc"}
Oct 31 00:56:49.558776 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.558742    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xqkgj" event={"ID":"75228856-5f8f-4d69-9c35-1012563b9ed1","Type":"ContainerStarted","Data":"d6ec5144bfe61c14b5ddd013178ebb414b4a06914e2b5d15ce268a9e7610feb1"}
Oct 31 00:56:49.559890 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.559867    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-78s6n" event={"ID":"70365afb-44c6-4aa3-8512-0f01c05ead6e","Type":"ContainerStarted","Data":"471a646149a8b7e24c48e7dfda9c82cef891b14bedf264401c632b14e2c2af9a"}
Oct 31 00:56:49.561679 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.561659    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rlr9b" event={"ID":"9d67aaf2-9211-4f6c-84d8-99aae85c28eb","Type":"ContainerStarted","Data":"4433fa5173b43f7aca56f533ca9b8dac187cf766ebe6f4cc4e7248584cdf2509"}
Oct 31 00:56:49.561776 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.561696    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rlr9b" event={"ID":"9d67aaf2-9211-4f6c-84d8-99aae85c28eb","Type":"ContainerStarted","Data":"492bfb45beeaf878b6babce8a9f5876765382e857e6f564a09ce6311b47832a6"}
Oct 31 00:56:49.660703 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.660667    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:49.677240 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.677169    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z" podStartSLOduration=1.677154686 podStartE2EDuration="1.677154686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:49.676638708 +0000 UTC m=+19.366521834" watchObservedRunningTime="2026-10-31 00:56:49.677154686 +0000 UTC m=+19.367037814"
Oct 31 00:56:49.745818 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.745786    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tw9gh"]
Oct 31 00:56:49.749125 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:56:49.749093    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e3e872_a804_480e_a267_c59f6ab82512.slice/crio-bab5f6a1d4c7510be548524b60158b0e44995b7ae3f5b3c03706ca3bd74104ad WatchSource:0}: Error finding container bab5f6a1d4c7510be548524b60158b0e44995b7ae3f5b3c03706ca3bd74104ad: Status 404 returned error can't find the container with id bab5f6a1d4c7510be548524b60158b0e44995b7ae3f5b3c03706ca3bd74104ad
Oct 31 00:56:49.910423 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:49.910337    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:50.567977 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:50.567938    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tw9gh" event={"ID":"83e3e872-a804-480e-a267-c59f6ab82512","Type":"ContainerStarted","Data":"bab5f6a1d4c7510be548524b60158b0e44995b7ae3f5b3c03706ca3bd74104ad"}
Oct 31 00:56:51.404865 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:51.404768    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-46s2r"
Oct 31 00:56:51.573195 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:51.573131    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xqkgj" event={"ID":"75228856-5f8f-4d69-9c35-1012563b9ed1","Type":"ContainerStarted","Data":"7aa3acb8347a6d9f830a9db0e214ee262f76f4476f06cf27cea7aa89d5d297b7"}
Oct 31 00:56:51.573686 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:51.573222    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xqkgj"
Oct 31 00:56:51.574816 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:51.574782    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tw9gh" event={"ID":"83e3e872-a804-480e-a267-c59f6ab82512","Type":"ContainerStarted","Data":"cef3d9de29a88354bfafdf016d00334b40e3e4c967e02e9043a5177bfba669c4"}
Oct 31 00:56:51.578719 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:51.578696    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xqkgj"
Oct 31 00:56:51.782402 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:51.782343    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xqkgj" podStartSLOduration=1.9277270340000001 podStartE2EDuration="3.782323953s" totalImagesPullingTime="1.854596919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:51.780768272 +0000 UTC m=+21.470651400" watchObservedRunningTime="2026-10-31 00:56:51.782323953 +0000 UTC m=+21.472207083"
Oct 31 00:56:51.893076 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:51.893018    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tw9gh" podStartSLOduration=2.236877109 podStartE2EDuration="3.892998471s" totalImagesPullingTime="1.656121362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:51.892689513 +0000 UTC m=+21.582572640" watchObservedRunningTime="2026-10-31 00:56:51.892998471 +0000 UTC m=+21.582881595"
Oct 31 00:56:52.194741 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:52.194658    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:52.881199 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:52.881135    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:53.081282 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:53.081250    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:53.283815 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:53.283781    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:53.502916 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:53.502877    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:53.715281 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:53.715195    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:54.748216 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:54.748167    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:54.939592 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:54.939551    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:58.219760 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:58.219722    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:58.436413 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:58.436348    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:58.720007 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:58.719954    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:59.144794 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.144754    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:59.183647 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:56:59.183614    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 00:56:59.330256 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.330207    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:59.554420 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.554370    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:59.766535 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.766504    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:59.827848 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.827777    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gvgm6"]
Oct 31 00:56:59.830152 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.830123    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:56:59.903284 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.903248    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 00:56:59.931272 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.931243    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 00:56:59.931462 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.931290    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 00:56:59.931462 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.931326    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 00:56:59.944085 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.944055    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-root\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:56:59.944582 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.944096    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-sys\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:56:59.944582 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.944128    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-metrics-client-ca\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:56:59.944582 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.944153    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7sp9\" (UniqueName: \"kubernetes.io/projected/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-kube-api-access-m7sp9\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:56:59.944582 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.944272    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:56:59.944582 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.944325    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-wtmp\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:56:59.944582 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.944379    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:56:59.944582 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.944403    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-textfile\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:56:59.944582 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.944418    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-tls\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:56:59.952973 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.952937    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 00:56:59.958016 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.957981    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:56:59.980778 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.980744    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7j8s4\""
Oct 31 00:56:59.980935 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:56:59.980834    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 00:57:00.045143 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.045097    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.045324 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.045148    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-wtmp\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.045324 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.045198    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.045324 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.045245    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-textfile\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.045324 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.045269    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-tls\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.045324 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.045325    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-root\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.045499 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.045351    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-sys\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.045499 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.045375    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-metrics-client-ca\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.045499 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.045398    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7sp9\" (UniqueName: \"kubernetes.io/projected/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-kube-api-access-m7sp9\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.046430 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.045686    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-sys\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.046430 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.045750    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-root\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.046430 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.046278    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-wtmp\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.046430 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.046380    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-textfile\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.046430 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.046428    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-metrics-client-ca\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.046746 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.046435    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.047994 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.047961    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.048436 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.048413    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-node-exporter-tls\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.103241 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.103146    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7sp9\" (UniqueName: \"kubernetes.io/projected/e087cf8c-2361-4702-9655-3ce6ee3c6f3d-kube-api-access-m7sp9\") pod \"node-exporter-gvgm6\" (UID: \"e087cf8c-2361-4702-9655-3ce6ee3c6f3d\") " pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.143810 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.143769    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gvgm6"
Oct 31 00:57:00.152621 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.152592    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:00.155030 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:57:00.155007    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode087cf8c_2361_4702_9655_3ce6ee3c6f3d.slice/crio-96499fb102f3f325ecf689e916e64b5766dc8368b4a51a17362ad065b5c59ed9 WatchSource:0}: Error finding container 96499fb102f3f325ecf689e916e64b5766dc8368b4a51a17362ad065b5c59ed9: Status 404 returned error can't find the container with id 96499fb102f3f325ecf689e916e64b5766dc8368b4a51a17362ad065b5c59ed9
Oct 31 00:57:00.346389 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.346360    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:00.438397 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.438327    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qhvwh"
Oct 31 00:57:00.612241 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:00.612202    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvgm6" event={"ID":"e087cf8c-2361-4702-9655-3ce6ee3c6f3d","Type":"ContainerStarted","Data":"96499fb102f3f325ecf689e916e64b5766dc8368b4a51a17362ad065b5c59ed9"}
Oct 31 00:57:01.255617 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:01.255575    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:01.531099 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:01.531058    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:02.144111 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:02.144074    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:02.342394 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:02.342366    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:02.534953 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:02.534908    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:02.621848 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:02.621808    4476 generic.go:356] "Generic (PLEG): container finished" podID="e087cf8c-2361-4702-9655-3ce6ee3c6f3d" containerID="c3502d6599a7b312df8a516c1e66a2c820e97c58609d4706a6dbd2376b2fbb68" exitCode=0
Oct 31 00:57:02.622024 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:02.621893    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvgm6" event={"ID":"e087cf8c-2361-4702-9655-3ce6ee3c6f3d","Type":"ContainerDied","Data":"c3502d6599a7b312df8a516c1e66a2c820e97c58609d4706a6dbd2376b2fbb68"}
Oct 31 00:57:02.873880 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:02.873840    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:03.061249 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:03.061213    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:03.628346 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:03.628312    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvgm6" event={"ID":"e087cf8c-2361-4702-9655-3ce6ee3c6f3d","Type":"ContainerStarted","Data":"ef1d7cf4b96ddd45bc3eafb17a7248bb447075da29dbf46137237215bdf0bd74"}
Oct 31 00:57:03.628770 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:03.628374    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gvgm6" event={"ID":"e087cf8c-2361-4702-9655-3ce6ee3c6f3d","Type":"ContainerStarted","Data":"a795bd6158da8961d3fbe280a6e69523abd560e3a7baf2ad3cc9e74d6233f245"}
Oct 31 00:57:03.793855 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:03.793806    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gvgm6" podStartSLOduration=2.566850137 podStartE2EDuration="4.793790044s" totalImagesPullingTime="2.127475015s" totalInitContainerRuntime="99.464892ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:03.793085949 +0000 UTC m=+33.482969077" watchObservedRunningTime="2026-10-31 00:57:03.793790044 +0000 UTC m=+33.483673171"
Oct 31 00:57:03.958368 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:03.958269    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:04.221693 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:04.221614    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:04.476473 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:04.476383    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:04.691707 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:04.691661    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:06.085840 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:06.085809    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:06.272389 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:06.272349    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:06.641218 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:06.641157    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-78s6n" event={"ID":"70365afb-44c6-4aa3-8512-0f01c05ead6e","Type":"ContainerStarted","Data":"46ecb274d49ff50e912b3e696f55b116be86e11cc5f1f484ae390a8a180bec69"}
Oct 31 00:57:06.641557 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:06.641342    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-78s6n"
Oct 31 00:57:06.656056 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:06.656021    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-78s6n"
Oct 31 00:57:06.973508 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:06.973401    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-78s6n" podStartSLOduration=2.415720355 podStartE2EDuration="18.97338428s" totalImagesPullingTime="16.557663925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:06.858626574 +0000 UTC m=+36.548509700" watchObservedRunningTime="2026-10-31 00:57:06.97338428 +0000 UTC m=+36.663267406"
Oct 31 00:57:08.892333 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:08.892297    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:09.758631 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:09.758597    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:09.958019 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:09.957984    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:10.421724 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:10.421689    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:10.573417 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:10.573388    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64949dc4d4-2tx9z"
Oct 31 00:57:10.631981 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:10.631943    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:10.960314 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:10.960278    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:11.183219 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:11.183161    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:11.374393 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:11.374363    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:13.358382 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:13.358341    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:13.569854 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:13.569820    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:14.487898 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:14.487865    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:14.702782 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:14.702752    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:14.901313 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:14.901275    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:16.053190 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:16.053146    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:16.249030 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:16.248997    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:16.467396 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:16.467315    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:20.628912 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:20.628877    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:20.805115 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:20.805058    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:22.942689 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:22.942653    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:23.126789 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:23.126752    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:24.754001 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:24.753917    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:24.935370 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:24.935335    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:29.180131 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:57:29.180097    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 00:57:38.607950 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:38.607914    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:38.793514 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:38.793476    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:39.758151 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:39.758115    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:39.933189 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:39.933145    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:52.977570 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:52.977537    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:53.479689 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:53.479654    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:53.708589 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:53.708558    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:57:53.897968 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:53.897931    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-dsx4h-8b9gr"]
Oct 31 00:57:53.898337 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:53.898312    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-dsx4h-8b9gr" podUID="506d546e-6187-4c6a-967b-09273e1fa785" containerName="insights-gathering" containerID="cri-o://2dd45acde478f2477a24827f5813867a2116a0de5904cf27d484b77616c415de" gracePeriod=30
Oct 31 00:57:54.139424 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.139400    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:57:54.235425 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.235361    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506d546e-6187-4c6a-967b-09273e1fa785-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/506d546e-6187-4c6a-967b-09273e1fa785-archives-path\") pod \"506d546e-6187-4c6a-967b-09273e1fa785\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") "
Oct 31 00:57:54.235425 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.235403    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506d546e-6187-4c6a-967b-09273e1fa785-kube-api-access-nrm7b\" (UniqueName: \"kubernetes.io/projected/506d546e-6187-4c6a-967b-09273e1fa785-kube-api-access-nrm7b\") pod \"506d546e-6187-4c6a-967b-09273e1fa785\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") "
Oct 31 00:57:54.235594 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.235434    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-trusted-ca-bundle\") pod \"506d546e-6187-4c6a-967b-09273e1fa785\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") "
Oct 31 00:57:54.235594 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.235466    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-service-ca-bundle\") pod \"506d546e-6187-4c6a-967b-09273e1fa785\" (UID: \"506d546e-6187-4c6a-967b-09273e1fa785\") "
Oct 31 00:57:54.235768 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.235734    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506d546e-6187-4c6a-967b-09273e1fa785-archives-path" pod "506d546e-6187-4c6a-967b-09273e1fa785" (UID: "506d546e-6187-4c6a-967b-09273e1fa785"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:54.235854 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.235830    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-trusted-ca-bundle" pod "506d546e-6187-4c6a-967b-09273e1fa785" (UID: "506d546e-6187-4c6a-967b-09273e1fa785"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:57:54.235917 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.235889    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-service-ca-bundle" pod "506d546e-6187-4c6a-967b-09273e1fa785" (UID: "506d546e-6187-4c6a-967b-09273e1fa785"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:57:54.237733 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.237712    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506d546e-6187-4c6a-967b-09273e1fa785-kube-api-access-nrm7b" pod "506d546e-6187-4c6a-967b-09273e1fa785" (UID: "506d546e-6187-4c6a-967b-09273e1fa785"). InnerVolumeSpecName "kube-api-access-nrm7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:57:54.336274 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.336233    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrm7b\" (UniqueName: \"kubernetes.io/projected/506d546e-6187-4c6a-967b-09273e1fa785-kube-api-access-nrm7b\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:57:54.336274 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.336271    4476 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-trusted-ca-bundle\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:57:54.336274 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.336281    4476 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506d546e-6187-4c6a-967b-09273e1fa785-service-ca-bundle\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:57:54.336274 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.336290    4476 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/506d546e-6187-4c6a-967b-09273e1fa785-archives-path\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:57:54.804570 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.804533    4476 generic.go:356] "Generic (PLEG): container finished" podID="506d546e-6187-4c6a-967b-09273e1fa785" containerID="2dd45acde478f2477a24827f5813867a2116a0de5904cf27d484b77616c415de" exitCode=2
Oct 31 00:57:54.804743 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.804585    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dsx4h-8b9gr"
Oct 31 00:57:54.804743 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.804586    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dsx4h-8b9gr" event={"ID":"506d546e-6187-4c6a-967b-09273e1fa785","Type":"ContainerDied","Data":"2dd45acde478f2477a24827f5813867a2116a0de5904cf27d484b77616c415de"}
Oct 31 00:57:54.804743 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.804662    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dsx4h-8b9gr" event={"ID":"506d546e-6187-4c6a-967b-09273e1fa785","Type":"ContainerDied","Data":"35e2c18924ca3bf73037fd4b2b57bcd54be0ecc893851fa01325cdef2cd9ffa2"}
Oct 31 00:57:54.804743 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.804682    4476 scope.go:122] "RemoveContainer" containerID="2dd45acde478f2477a24827f5813867a2116a0de5904cf27d484b77616c415de"
Oct 31 00:57:54.812265 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.812243    4476 scope.go:122] "RemoveContainer" containerID="2dd45acde478f2477a24827f5813867a2116a0de5904cf27d484b77616c415de"
Oct 31 00:57:54.812533 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:57:54.812515    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd45acde478f2477a24827f5813867a2116a0de5904cf27d484b77616c415de\": container with ID starting with 2dd45acde478f2477a24827f5813867a2116a0de5904cf27d484b77616c415de not found: ID does not exist" containerID="2dd45acde478f2477a24827f5813867a2116a0de5904cf27d484b77616c415de"
Oct 31 00:57:54.812599 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.812541    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd45acde478f2477a24827f5813867a2116a0de5904cf27d484b77616c415de"} err="failed to get container status \"2dd45acde478f2477a24827f5813867a2116a0de5904cf27d484b77616c415de\": rpc error: code = NotFound desc = could not find container \"2dd45acde478f2477a24827f5813867a2116a0de5904cf27d484b77616c415de\": container with ID starting with 2dd45acde478f2477a24827f5813867a2116a0de5904cf27d484b77616c415de not found: ID does not exist"
Oct 31 00:57:54.964812 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:54.964787    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-dsx4h-8b9gr"]
Oct 31 00:57:55.014030 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:55.014010    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-dsx4h-8b9gr"]
Oct 31 00:57:56.451366 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:56.451338    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="506d546e-6187-4c6a-967b-09273e1fa785" path="/var/lib/kubelet/pods/506d546e-6187-4c6a-967b-09273e1fa785/volumes"
Oct 31 00:57:59.179476 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:57:59.179434    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 00:57:59.330345 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.330311    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-655b45697-bp2zh"]
Oct 31 00:57:59.330628 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.330612    4476 cpu_manager.go:414] "containerMap: removing container" podUID="506d546e-6187-4c6a-967b-09273e1fa785" containerName="insights-gathering"
Oct 31 00:57:59.330706 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.330630    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="506d546e-6187-4c6a-967b-09273e1fa785" containerName="insights-gathering"
Oct 31 00:57:59.330706 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.330694    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="506d546e-6187-4c6a-967b-09273e1fa785" containerName="insights-gathering"
Oct 31 00:57:59.332888 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.332861    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.370495 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.370468    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b8249f4-55b9-40a2-9d79-2265aafb9c78-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.370616 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.370500    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b8249f4-55b9-40a2-9d79-2265aafb9c78-telemeter-client-tls\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.370616 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.370517    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b8249f4-55b9-40a2-9d79-2265aafb9c78-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.370616 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.370545    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nhk7\" (UniqueName: \"kubernetes.io/projected/7b8249f4-55b9-40a2-9d79-2265aafb9c78-kube-api-access-8nhk7\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.370732 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.370654    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b8249f4-55b9-40a2-9d79-2265aafb9c78-serving-certs-ca-bundle\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.370732 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.370703    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b8249f4-55b9-40a2-9d79-2265aafb9c78-secret-telemeter-client\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.370804 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.370748    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b8249f4-55b9-40a2-9d79-2265aafb9c78-federate-client-tls\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.370804 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.370777    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b8249f4-55b9-40a2-9d79-2265aafb9c78-metrics-client-ca\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.384924 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.384898    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 00:57:59.435447 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.435379    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-655b45697-bp2zh"]
Oct 31 00:57:59.471355 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.471332    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b8249f4-55b9-40a2-9d79-2265aafb9c78-serving-certs-ca-bundle\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.471433 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.471370    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b8249f4-55b9-40a2-9d79-2265aafb9c78-secret-telemeter-client\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.471433 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.471390    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b8249f4-55b9-40a2-9d79-2265aafb9c78-federate-client-tls\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.471433 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.471409    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b8249f4-55b9-40a2-9d79-2265aafb9c78-metrics-client-ca\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.471619 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.471591    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b8249f4-55b9-40a2-9d79-2265aafb9c78-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.471665 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.471649    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b8249f4-55b9-40a2-9d79-2265aafb9c78-telemeter-client-tls\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.471706 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.471679    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b8249f4-55b9-40a2-9d79-2265aafb9c78-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.471741 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.471721    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nhk7\" (UniqueName: \"kubernetes.io/projected/7b8249f4-55b9-40a2-9d79-2265aafb9c78-kube-api-access-8nhk7\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.472048 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.472033    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b8249f4-55b9-40a2-9d79-2265aafb9c78-metrics-client-ca\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.472108 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.472091    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b8249f4-55b9-40a2-9d79-2265aafb9c78-serving-certs-ca-bundle\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.533771 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.533739    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 00:57:59.533771 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.533742    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 00:57:59.533771 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.533771    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-g8244\""
Oct 31 00:57:59.533941 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.533745    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 00:57:59.533941 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.533745    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 00:57:59.533941 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.533882    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 00:57:59.536239 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.536213    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nhk7\" (UniqueName: \"kubernetes.io/projected/7b8249f4-55b9-40a2-9d79-2265aafb9c78-kube-api-access-8nhk7\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.543029 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.543003    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b8249f4-55b9-40a2-9d79-2265aafb9c78-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.544131 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.544107    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b8249f4-55b9-40a2-9d79-2265aafb9c78-federate-client-tls\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.544278 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.544257    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b8249f4-55b9-40a2-9d79-2265aafb9c78-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.544404 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.544389    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b8249f4-55b9-40a2-9d79-2265aafb9c78-secret-telemeter-client\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.544528 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.544510    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b8249f4-55b9-40a2-9d79-2265aafb9c78-telemeter-client-tls\") pod \"telemeter-client-655b45697-bp2zh\" (UID: \"7b8249f4-55b9-40a2-9d79-2265aafb9c78\") " pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:57:59.644558 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:57:59.644514    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-655b45697-bp2zh"
Oct 31 00:58:00.081810 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:00.081777    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-655b45697-bp2zh"]
Oct 31 00:58:00.084799 ip-10-0-139-174 kubenswrapper[4476]: W1031 00:58:00.084773    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8249f4_55b9_40a2_9d79_2265aafb9c78.slice/crio-a4442cb99e8ffab9388c216ae8a41fa68908b42d342d18edf864c296beefb3fe WatchSource:0}: Error finding container a4442cb99e8ffab9388c216ae8a41fa68908b42d342d18edf864c296beefb3fe: Status 404 returned error can't find the container with id a4442cb99e8ffab9388c216ae8a41fa68908b42d342d18edf864c296beefb3fe
Oct 31 00:58:00.824720 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:00.824682    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655b45697-bp2zh" event={"ID":"7b8249f4-55b9-40a2-9d79-2265aafb9c78","Type":"ContainerStarted","Data":"a4442cb99e8ffab9388c216ae8a41fa68908b42d342d18edf864c296beefb3fe"}
Oct 31 00:58:02.832325 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:02.832290    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655b45697-bp2zh" event={"ID":"7b8249f4-55b9-40a2-9d79-2265aafb9c78","Type":"ContainerStarted","Data":"572a015842e8bf4bdc750b4bce0dd0fbf1bfa1b3c80184760f39cd0c50e34658"}
Oct 31 00:58:03.837452 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:03.837415    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655b45697-bp2zh" event={"ID":"7b8249f4-55b9-40a2-9d79-2265aafb9c78","Type":"ContainerStarted","Data":"9992de07cb3fefd206a6d2a410fccabe4404787918f8077b94aa305a2c2d035d"}
Oct 31 00:58:03.837876 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:03.837475    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655b45697-bp2zh" event={"ID":"7b8249f4-55b9-40a2-9d79-2265aafb9c78","Type":"ContainerStarted","Data":"940d0d4fe31ef70b22a21807b6a08ffb691463d2fab43708c528f817155b5c53"}
Oct 31 00:58:04.056972 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:04.056916    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-655b45697-bp2zh" podStartSLOduration=1.5254528139999999 podStartE2EDuration="5.056902968s" totalImagesPullingTime="3.531450154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:04.056586353 +0000 UTC m=+93.746469479" watchObservedRunningTime="2026-10-31 00:58:04.056902968 +0000 UTC m=+93.746786094"
Oct 31 00:58:04.759264 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:04.759229    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:04.948959 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:04.948929    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:05.148232 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:05.148198    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:05.326631 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:05.326600    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:16.706050 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:16.706016    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:29.180088 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:58:29.180048    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 00:58:31.276869 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:31.276832    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:31.454388 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:31.454359    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:34.811185 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:34.811153    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:34.991994 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:34.991955    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:35.198097 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:35.198003    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:35.371671 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:35.371641    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:35.769904 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:35.769878    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:36.544294 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:36.544262    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:43.902782 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:43.902700    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:44.086110 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:44.086081    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:54.794703 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:54.794668    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:54.969883 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:58:54.969853    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:58:59.180500 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:58:59.180462    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 00:59:17.020942 ip-10-0-139-174 kubenswrapper[4476]: I1031 00:59:17.020911    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 00:59:29.178845 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:59:29.178798    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 00:59:59.179561 ip-10-0-139-174 kubenswrapper[4476]: E1031 00:59:59.179518    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:00:12.454625 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:00:12.454549    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:00:17.322878 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:00:17.322847    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:00:24.165423 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:00:24.165389    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:00:29.180103 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:00:29.180064    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:00:56.909987 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:00:56.909946    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:00:59.182440 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:00:59.182397    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:00:59.808119 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:00:59.808083    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:00.107486 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:00.107403    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:01.481405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.481369    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"]
Oct 31 01:01:01.484514 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.484498    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"
Oct 31 01:01:01.586486 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.586456    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-tkswg\""
Oct 31 01:01:01.586486 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.586464    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 01:01:01.587509 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.587488    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"]
Oct 31 01:01:01.599876 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.599854    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwbm8\" (UniqueName: \"kubernetes.io/projected/60c72f80-44c8-4da5-b068-4e2680d828fb-kube-api-access-bwbm8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk\" (UID: \"60c72f80-44c8-4da5-b068-4e2680d828fb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"
Oct 31 01:01:01.599995 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.599909    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60c72f80-44c8-4da5-b068-4e2680d828fb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk\" (UID: \"60c72f80-44c8-4da5-b068-4e2680d828fb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"
Oct 31 01:01:01.599995 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.599970    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c72f80-44c8-4da5-b068-4e2680d828fb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk\" (UID: \"60c72f80-44c8-4da5-b068-4e2680d828fb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"
Oct 31 01:01:01.619235 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.619209    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 01:01:01.648994 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.648966    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"]
Oct 31 01:01:01.652094 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.652080    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.700914 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.700876    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60c72f80-44c8-4da5-b068-4e2680d828fb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk\" (UID: \"60c72f80-44c8-4da5-b068-4e2680d828fb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"
Oct 31 01:01:01.701107 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.700921    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c72f80-44c8-4da5-b068-4e2680d828fb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk\" (UID: \"60c72f80-44c8-4da5-b068-4e2680d828fb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"
Oct 31 01:01:01.701107 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.700961    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwbm8\" (UniqueName: \"kubernetes.io/projected/60c72f80-44c8-4da5-b068-4e2680d828fb-kube-api-access-bwbm8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk\" (UID: \"60c72f80-44c8-4da5-b068-4e2680d828fb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"
Oct 31 01:01:01.701394 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.701368    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60c72f80-44c8-4da5-b068-4e2680d828fb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk\" (UID: \"60c72f80-44c8-4da5-b068-4e2680d828fb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"
Oct 31 01:01:01.701452 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.701375    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c72f80-44c8-4da5-b068-4e2680d828fb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk\" (UID: \"60c72f80-44c8-4da5-b068-4e2680d828fb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"
Oct 31 01:01:01.706291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.706275    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 01:01:01.715006 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.714922    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 01:01:01.715006 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.714967    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 01:01:01.715313 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.715293    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dtmj7\""
Oct 31 01:01:01.716075 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.716053    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 01:01:01.716220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.716094    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 01:01:01.716400 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.716368    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 01:01:01.716615 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.716593    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"]
Oct 31 01:01:01.741074 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.741007    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:01.772327 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.772299    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwbm8\" (UniqueName: \"kubernetes.io/projected/60c72f80-44c8-4da5-b068-4e2680d828fb-kube-api-access-bwbm8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk\" (UID: \"60c72f80-44c8-4da5-b068-4e2680d828fb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"
Oct 31 01:01:01.800237 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.800213    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"
Oct 31 01:01:01.801465 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.801446    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.801538 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.801478    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.801577 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.801564    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.801609 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.801588    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.801649 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.801610    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.801649 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.801629    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vddkm\" (UniqueName: \"kubernetes.io/projected/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-kube-api-access-vddkm\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.801767 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.801755    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.903382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.903344    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.903672 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.903654    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.903795 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.903781    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.903912 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.903895    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.904013 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.904000    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vddkm\" (UniqueName: \"kubernetes.io/projected/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-kube-api-access-vddkm\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.904125 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.904113    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.904276 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.904260    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.906146 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.906114    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.906970 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.906665    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.908137 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.907274    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.909289 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.909267    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.910236 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.910211    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.966554 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.966525    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vddkm\" (UniqueName: \"kubernetes.io/projected/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-kube-api-access-vddkm\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.967251 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.967232    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-phf8w\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:01.994746 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:01.994685    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:02.186240 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:02.186208    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"]
Oct 31 01:01:02.190124 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:01:02.190096    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c72f80_44c8_4da5_b068_4e2680d828fb.slice/crio-521a48de1a2c6170ed3b0f013ace52e36870a6d56c22708a17f70ddd4e7076d4 WatchSource:0}: Error finding container 521a48de1a2c6170ed3b0f013ace52e36870a6d56c22708a17f70ddd4e7076d4: Status 404 returned error can't find the container with id 521a48de1a2c6170ed3b0f013ace52e36870a6d56c22708a17f70ddd4e7076d4
Oct 31 01:01:02.263756 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:02.263672    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:02.385271 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:02.385238    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk" event={"ID":"60c72f80-44c8-4da5-b068-4e2680d828fb","Type":"ContainerStarted","Data":"521a48de1a2c6170ed3b0f013ace52e36870a6d56c22708a17f70ddd4e7076d4"}
Oct 31 01:01:02.689260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:02.689228    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"]
Oct 31 01:01:02.692320 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:01:02.692297    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fd2b99_c816_4a1a_9142_a2ec2d8754ef.slice/crio-34ff2509c60db21056e9e9c83a01d53a44742b744ecb8c90d4110a0e312850ae WatchSource:0}: Error finding container 34ff2509c60db21056e9e9c83a01d53a44742b744ecb8c90d4110a0e312850ae: Status 404 returned error can't find the container with id 34ff2509c60db21056e9e9c83a01d53a44742b744ecb8c90d4110a0e312850ae
Oct 31 01:01:03.099350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:03.099320    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:03.390242 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:03.390146    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w" event={"ID":"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef","Type":"ContainerStarted","Data":"34ff2509c60db21056e9e9c83a01d53a44742b744ecb8c90d4110a0e312850ae"}
Oct 31 01:01:07.905820 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:07.905779    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:01:07.906195 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:07.905866    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:01:08.413359 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:08.413323    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w" event={"ID":"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef","Type":"ContainerStarted","Data":"e2139180c1d661ddeac3ee3ee6bd7286ca69d093c9c89acbe4745c8e7189cf58"}
Oct 31 01:01:08.413556 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:08.413479    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:08.415481 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:08.415451    4476 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-phf8w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:01:08.415621 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:08.415508    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w" podUID="e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:01:08.620477 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:08.620419    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w" podStartSLOduration=2.409786403 podStartE2EDuration="7.620401392s" totalImagesPullingTime="5.210614989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:08.619378702 +0000 UTC m=+278.309261830" watchObservedRunningTime="2026-10-31 01:01:08.620401392 +0000 UTC m=+278.310284520"
Oct 31 01:01:09.362007 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:09.361978    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:09.417797 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:09.417767    4476 generic.go:356] "Generic (PLEG): container finished" podID="60c72f80-44c8-4da5-b068-4e2680d828fb" containerID="3bdbf595f09befec5a38a1ce20f78bb80e60d39755c0d93326ae1edd6ece1dc3" exitCode=0
Oct 31 01:01:09.417955 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:09.417859    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk" event={"ID":"60c72f80-44c8-4da5-b068-4e2680d828fb","Type":"ContainerDied","Data":"3bdbf595f09befec5a38a1ce20f78bb80e60d39755c0d93326ae1edd6ece1dc3"}
Oct 31 01:01:09.418929 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:09.418690    4476 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-phf8w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:01:09.418929 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:09.418731    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w" podUID="e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:01:09.551549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:09.551511    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:10.387019 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:10.386914    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:10.575939 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:10.575899    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:10.761651 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:10.761572    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:12.418857 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:12.418821    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:01:13.436906 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:13.436870    4476 generic.go:356] "Generic (PLEG): container finished" podID="60c72f80-44c8-4da5-b068-4e2680d828fb" containerID="4ffbff2a15ca3f3e72bfa18293e661743ebce1b34c49a861939c65e24faead1b" exitCode=0
Oct 31 01:01:13.437296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:13.436935    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk" event={"ID":"60c72f80-44c8-4da5-b068-4e2680d828fb","Type":"ContainerDied","Data":"4ffbff2a15ca3f3e72bfa18293e661743ebce1b34c49a861939c65e24faead1b"}
Oct 31 01:01:14.604554 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:14.604519    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:17.730434 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:17.730356    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:21.467206 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:21.467161    4476 generic.go:356] "Generic (PLEG): container finished" podID="60c72f80-44c8-4da5-b068-4e2680d828fb" containerID="8f0771759b711cf06fdfa423611a2ebc262f988b006d2ec98f4cad59b50c6046" exitCode=0
Oct 31 01:01:21.467561 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:21.467209    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk" event={"ID":"60c72f80-44c8-4da5-b068-4e2680d828fb","Type":"ContainerDied","Data":"8f0771759b711cf06fdfa423611a2ebc262f988b006d2ec98f4cad59b50c6046"}
Oct 31 01:01:22.589366 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:22.589341    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"
Oct 31 01:01:22.669834 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:22.669800    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c72f80-44c8-4da5-b068-4e2680d828fb-util\" (UniqueName: \"kubernetes.io/empty-dir/60c72f80-44c8-4da5-b068-4e2680d828fb-util\") pod \"60c72f80-44c8-4da5-b068-4e2680d828fb\" (UID: \"60c72f80-44c8-4da5-b068-4e2680d828fb\") "
Oct 31 01:01:22.669993 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:22.669867    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c72f80-44c8-4da5-b068-4e2680d828fb-kube-api-access-bwbm8\" (UniqueName: \"kubernetes.io/projected/60c72f80-44c8-4da5-b068-4e2680d828fb-kube-api-access-bwbm8\") pod \"60c72f80-44c8-4da5-b068-4e2680d828fb\" (UID: \"60c72f80-44c8-4da5-b068-4e2680d828fb\") "
Oct 31 01:01:22.669993 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:22.669930    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c72f80-44c8-4da5-b068-4e2680d828fb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c72f80-44c8-4da5-b068-4e2680d828fb-bundle\") pod \"60c72f80-44c8-4da5-b068-4e2680d828fb\" (UID: \"60c72f80-44c8-4da5-b068-4e2680d828fb\") "
Oct 31 01:01:22.670381 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:22.670352    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c72f80-44c8-4da5-b068-4e2680d828fb-bundle" pod "60c72f80-44c8-4da5-b068-4e2680d828fb" (UID: "60c72f80-44c8-4da5-b068-4e2680d828fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:22.672286 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:22.672260    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c72f80-44c8-4da5-b068-4e2680d828fb-kube-api-access-bwbm8" pod "60c72f80-44c8-4da5-b068-4e2680d828fb" (UID: "60c72f80-44c8-4da5-b068-4e2680d828fb"). InnerVolumeSpecName "kube-api-access-bwbm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:22.674234 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:22.674211    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c72f80-44c8-4da5-b068-4e2680d828fb-util" pod "60c72f80-44c8-4da5-b068-4e2680d828fb" (UID: "60c72f80-44c8-4da5-b068-4e2680d828fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:01:22.771523 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:22.771442    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c72f80-44c8-4da5-b068-4e2680d828fb-bundle\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:01:22.771523 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:22.771472    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60c72f80-44c8-4da5-b068-4e2680d828fb-util\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:01:22.771523 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:22.771483    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwbm8\" (UniqueName: \"kubernetes.io/projected/60c72f80-44c8-4da5-b068-4e2680d828fb-kube-api-access-bwbm8\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:01:23.475199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:23.475149    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk" event={"ID":"60c72f80-44c8-4da5-b068-4e2680d828fb","Type":"ContainerDied","Data":"521a48de1a2c6170ed3b0f013ace52e36870a6d56c22708a17f70ddd4e7076d4"}
Oct 31 01:01:23.475199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:23.475167    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hsbqk"
Oct 31 01:01:23.475411 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:23.475211    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="521a48de1a2c6170ed3b0f013ace52e36870a6d56c22708a17f70ddd4e7076d4"
Oct 31 01:01:27.703752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:27.703720    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:29.180158 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:01:29.180121    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:01:45.135458 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:45.135372    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:45.501470 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:45.501398    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:46.087521 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.087483    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:46.336324 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.336288    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:46.456765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.456678    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"]
Oct 31 01:01:46.457004 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.456987    4476 cpu_manager.go:414] "containerMap: removing container" podUID="60c72f80-44c8-4da5-b068-4e2680d828fb" containerName="pull"
Oct 31 01:01:46.457085 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.457007    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c72f80-44c8-4da5-b068-4e2680d828fb" containerName="pull"
Oct 31 01:01:46.457085 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.457034    4476 cpu_manager.go:414] "containerMap: removing container" podUID="60c72f80-44c8-4da5-b068-4e2680d828fb" containerName="util"
Oct 31 01:01:46.457085 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.457042    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c72f80-44c8-4da5-b068-4e2680d828fb" containerName="util"
Oct 31 01:01:46.457085 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.457062    4476 cpu_manager.go:414] "containerMap: removing container" podUID="60c72f80-44c8-4da5-b068-4e2680d828fb" containerName="extract"
Oct 31 01:01:46.457085 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.457070    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c72f80-44c8-4da5-b068-4e2680d828fb" containerName="extract"
Oct 31 01:01:46.457362 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.457139    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c72f80-44c8-4da5-b068-4e2680d828fb" containerName="extract"
Oct 31 01:01:46.458878 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.458855    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.533695 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.533667    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ca750c5-f8bb-4a8d-9579-71b6731da112-tmp\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.533823 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.533701    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ca750c5-f8bb-4a8d-9579-71b6731da112-metrics-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.533823 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.533728    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ca750c5-f8bb-4a8d-9579-71b6731da112-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.533902 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.533835    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ca750c5-f8bb-4a8d-9579-71b6731da112-manifests\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.533902 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.533865    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx6x7\" (UniqueName: \"kubernetes.io/projected/4ca750c5-f8bb-4a8d-9579-71b6731da112-kube-api-access-bx6x7\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.533902 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.533896    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ca750c5-f8bb-4a8d-9579-71b6731da112-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.542817 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.542796    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:46.635241 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.635209    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ca750c5-f8bb-4a8d-9579-71b6731da112-manifests\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.635241 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.635244    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx6x7\" (UniqueName: \"kubernetes.io/projected/4ca750c5-f8bb-4a8d-9579-71b6731da112-kube-api-access-bx6x7\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.635464 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.635273    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ca750c5-f8bb-4a8d-9579-71b6731da112-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.635464 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.635303    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ca750c5-f8bb-4a8d-9579-71b6731da112-tmp\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.635464 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.635322    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ca750c5-f8bb-4a8d-9579-71b6731da112-metrics-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.635464 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.635340    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ca750c5-f8bb-4a8d-9579-71b6731da112-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.635679 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.635649    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ca750c5-f8bb-4a8d-9579-71b6731da112-manifests\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.637523 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.637498    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ca750c5-f8bb-4a8d-9579-71b6731da112-tmp\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.638841 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.638822    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 01:01:46.638900 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.638860    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wffsw\""
Oct 31 01:01:46.638941 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.638912    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 01:01:46.638985 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.638966    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 01:01:46.639054 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.639037    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 01:01:46.639124 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.639111    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 01:01:46.648495 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.648474    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ca750c5-f8bb-4a8d-9579-71b6731da112-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.648607 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.648587    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ca750c5-f8bb-4a8d-9579-71b6731da112-metrics-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.648650 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.648626    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ca750c5-f8bb-4a8d-9579-71b6731da112-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.688597 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.688567    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"]
Oct 31 01:01:46.693739 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.693711    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx6x7\" (UniqueName: \"kubernetes.io/projected/4ca750c5-f8bb-4a8d-9579-71b6731da112-kube-api-access-bx6x7\") pod \"opendatahub-operator-controller-manager-7fcdb566-czqtv\" (UID: \"4ca750c5-f8bb-4a8d-9579-71b6731da112\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.733402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.733332    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:46.770555 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.770534    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:46.973700 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:46.973662    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:47.180169 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:47.180136    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:47.258460 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:47.258401    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"]
Oct 31 01:01:47.260635 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:01:47.260598    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca750c5_f8bb_4a8d_9579_71b6731da112.slice/crio-47be397408c4aad8b7f534dfcb65aa6845ade399779be342656ea77541450967 WatchSource:0}: Error finding container 47be397408c4aad8b7f534dfcb65aa6845ade399779be342656ea77541450967: Status 404 returned error can't find the container with id 47be397408c4aad8b7f534dfcb65aa6845ade399779be342656ea77541450967
Oct 31 01:01:47.262279 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:47.262261    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:01:47.553239 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:47.553205    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv" event={"ID":"4ca750c5-f8bb-4a8d-9579-71b6731da112","Type":"ContainerStarted","Data":"47be397408c4aad8b7f534dfcb65aa6845ade399779be342656ea77541450967"}
Oct 31 01:01:48.913081 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:48.912986    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:50.233215 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:50.233161    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:50.455819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:50.455762    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:51.569541 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:51.569497    4476 generic.go:356] "Generic (PLEG): container finished" podID="4ca750c5-f8bb-4a8d-9579-71b6731da112" containerID="992f9e498e908de3203908a1b43fb1f6e67bc943fcd17fac53c3a8aebcede96b" exitCode=0
Oct 31 01:01:51.569966 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:51.569581    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv" event={"ID":"4ca750c5-f8bb-4a8d-9579-71b6731da112","Type":"ContainerDied","Data":"992f9e498e908de3203908a1b43fb1f6e67bc943fcd17fac53c3a8aebcede96b"}
Oct 31 01:01:52.575451 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:52.575419    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv" event={"ID":"4ca750c5-f8bb-4a8d-9579-71b6731da112","Type":"ContainerStarted","Data":"da207c434ac067549cc238b7c31d15dcbc61ad4564e5fa00c3ca8d5930a3279b"}
Oct 31 01:01:52.575853 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:52.575566    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:01:52.815282 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:52.815238    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv" podStartSLOduration=2.146341257 podStartE2EDuration="6.815224085s" totalImagesPullingTime="4.243792673s" totalInitContainerRuntime="425.090155ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:52.765758088 +0000 UTC m=+322.455641215" watchObservedRunningTime="2026-10-31 01:01:52.815224085 +0000 UTC m=+322.505107211"
Oct 31 01:01:53.904329 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:53.904295    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:54.136553 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:54.136522    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:54.636369 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:54.636340    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:55.249601 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:55.249544    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:56.797362 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:56.797330    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:57.063431 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:57.063349    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:57.688132 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:57.688101    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:57.971614 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:57.971515    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:58.697649 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:58.697619    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:01:59.179354 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:01:59.179319    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:01:59.825477 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:01:59.825444    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:00.060563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.060527    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"]
Oct 31 01:02:00.064585 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.064569    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"
Oct 31 01:02:00.118910 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.118846    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 01:02:00.119019 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.118942    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-tkswg\""
Oct 31 01:02:00.135124 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.135102    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de5a8a39-81f2-4538-99d6-6da9bc70e138-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb\" (UID: \"de5a8a39-81f2-4538-99d6-6da9bc70e138\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"
Oct 31 01:02:00.135259 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.135134    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cpdc\" (UniqueName: \"kubernetes.io/projected/de5a8a39-81f2-4538-99d6-6da9bc70e138-kube-api-access-6cpdc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb\" (UID: \"de5a8a39-81f2-4538-99d6-6da9bc70e138\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"
Oct 31 01:02:00.135259 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.135167    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de5a8a39-81f2-4538-99d6-6da9bc70e138-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb\" (UID: \"de5a8a39-81f2-4538-99d6-6da9bc70e138\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"
Oct 31 01:02:00.168521 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.168502    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 01:02:00.169662 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.169642    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"]
Oct 31 01:02:00.236371 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.236340    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cpdc\" (UniqueName: \"kubernetes.io/projected/de5a8a39-81f2-4538-99d6-6da9bc70e138-kube-api-access-6cpdc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb\" (UID: \"de5a8a39-81f2-4538-99d6-6da9bc70e138\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"
Oct 31 01:02:00.236703 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.236393    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de5a8a39-81f2-4538-99d6-6da9bc70e138-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb\" (UID: \"de5a8a39-81f2-4538-99d6-6da9bc70e138\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"
Oct 31 01:02:00.236703 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.236436    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de5a8a39-81f2-4538-99d6-6da9bc70e138-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb\" (UID: \"de5a8a39-81f2-4538-99d6-6da9bc70e138\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"
Oct 31 01:02:00.236775 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.236730    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de5a8a39-81f2-4538-99d6-6da9bc70e138-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb\" (UID: \"de5a8a39-81f2-4538-99d6-6da9bc70e138\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"
Oct 31 01:02:00.236808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.236774    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de5a8a39-81f2-4538-99d6-6da9bc70e138-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb\" (UID: \"de5a8a39-81f2-4538-99d6-6da9bc70e138\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"
Oct 31 01:02:00.295880 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.295853    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cpdc\" (UniqueName: \"kubernetes.io/projected/de5a8a39-81f2-4538-99d6-6da9bc70e138-kube-api-access-6cpdc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb\" (UID: \"de5a8a39-81f2-4538-99d6-6da9bc70e138\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"
Oct 31 01:02:00.376454 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.376391    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"
Oct 31 01:02:00.583296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.583264    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:00.747947 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:00.747876    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"]
Oct 31 01:02:00.779788 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:02:00.779759    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5a8a39_81f2_4538_99d6_6da9bc70e138.slice/crio-7c6e20bdec16fb44e291831d7385959d3d19807233bd17ae90c71d615a92cfce WatchSource:0}: Error finding container 7c6e20bdec16fb44e291831d7385959d3d19807233bd17ae90c71d615a92cfce: Status 404 returned error can't find the container with id 7c6e20bdec16fb44e291831d7385959d3d19807233bd17ae90c71d615a92cfce
Oct 31 01:02:01.605352 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:01.605272    4476 generic.go:356] "Generic (PLEG): container finished" podID="de5a8a39-81f2-4538-99d6-6da9bc70e138" containerID="7339b99a5747170770a8b2b4de3bf058eb8f5895dc5494a04538b1944dcf8312" exitCode=0
Oct 31 01:02:01.605352 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:01.605304    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb" event={"ID":"de5a8a39-81f2-4538-99d6-6da9bc70e138","Type":"ContainerDied","Data":"7339b99a5747170770a8b2b4de3bf058eb8f5895dc5494a04538b1944dcf8312"}
Oct 31 01:02:01.605352 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:01.605336    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb" event={"ID":"de5a8a39-81f2-4538-99d6-6da9bc70e138","Type":"ContainerStarted","Data":"7c6e20bdec16fb44e291831d7385959d3d19807233bd17ae90c71d615a92cfce"}
Oct 31 01:02:02.610311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:02.610277    4476 generic.go:356] "Generic (PLEG): container finished" podID="de5a8a39-81f2-4538-99d6-6da9bc70e138" containerID="2028117864a841d18c7632b5476cf58eac6c32826450b7a37f99b855eba74d95" exitCode=0
Oct 31 01:02:02.610745 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:02.610371    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb" event={"ID":"de5a8a39-81f2-4538-99d6-6da9bc70e138","Type":"ContainerDied","Data":"2028117864a841d18c7632b5476cf58eac6c32826450b7a37f99b855eba74d95"}
Oct 31 01:02:02.612712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:02.612686    4476 generic.go:356] "Generic (PLEG): container finished" podID="4ca750c5-f8bb-4a8d-9579-71b6731da112" containerID="da207c434ac067549cc238b7c31d15dcbc61ad4564e5fa00c3ca8d5930a3279b" exitCode=1
Oct 31 01:02:02.612791 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:02.612715    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv" event={"ID":"4ca750c5-f8bb-4a8d-9579-71b6731da112","Type":"ContainerDied","Data":"da207c434ac067549cc238b7c31d15dcbc61ad4564e5fa00c3ca8d5930a3279b"}
Oct 31 01:02:02.613071 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:02.613058    4476 scope.go:122] "RemoveContainer" containerID="da207c434ac067549cc238b7c31d15dcbc61ad4564e5fa00c3ca8d5930a3279b"
Oct 31 01:02:03.076303 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:03.076247    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:03.618807 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:03.618764    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb" event={"ID":"de5a8a39-81f2-4538-99d6-6da9bc70e138","Type":"ContainerStarted","Data":"8372953f7ca5ab409b344f231c0538e385f51c1dc17aee21f92f010af605b88f"}
Oct 31 01:02:03.625083 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:03.625053    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv" event={"ID":"4ca750c5-f8bb-4a8d-9579-71b6731da112","Type":"ContainerStarted","Data":"0a06db55bc73d274d67f11ef915d8fa825379c3f4defb48ceca477179d4938c2"}
Oct 31 01:02:03.625354 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:03.625339    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:02:03.750169 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:03.750087    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb" podStartSLOduration=3.75007438 podStartE2EDuration="3.75007438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:03.748160129 +0000 UTC m=+333.438043257" watchObservedRunningTime="2026-10-31 01:02:03.75007438 +0000 UTC m=+333.439957545"
Oct 31 01:02:04.124011 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:04.123977    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:04.337193 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:04.337151    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:04.729767 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:04.729739    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:04.971203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:04.971150    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:05.243240 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:05.243210    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:08.209718 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:08.209689    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:10.274643 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:10.274614    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:11.723095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:11.723063    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:12.916713 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:12.916680    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:13.499846 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:13.499813    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:13.659986 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:13.659954    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:02:13.660764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:13.660733    4476 generic.go:356] "Generic (PLEG): container finished" podID="de5a8a39-81f2-4538-99d6-6da9bc70e138" containerID="8372953f7ca5ab409b344f231c0538e385f51c1dc17aee21f92f010af605b88f" exitCode=1
Oct 31 01:02:13.660889 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:13.660763    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb" event={"ID":"de5a8a39-81f2-4538-99d6-6da9bc70e138","Type":"ContainerDied","Data":"8372953f7ca5ab409b344f231c0538e385f51c1dc17aee21f92f010af605b88f"}
Oct 31 01:02:13.744337 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:13.744306    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:14.631158 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:14.631107    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-czqtv"
Oct 31 01:02:14.790482 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:14.790463    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:02:14.793275 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:14.793258    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"
Oct 31 01:02:14.948926 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:14.948851    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5a8a39-81f2-4538-99d6-6da9bc70e138-kube-api-access-6cpdc\" (UniqueName: \"kubernetes.io/projected/de5a8a39-81f2-4538-99d6-6da9bc70e138-kube-api-access-6cpdc\") pod \"de5a8a39-81f2-4538-99d6-6da9bc70e138\" (UID: \"de5a8a39-81f2-4538-99d6-6da9bc70e138\") "
Oct 31 01:02:14.948926 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:14.948912    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5a8a39-81f2-4538-99d6-6da9bc70e138-util\" (UniqueName: \"kubernetes.io/empty-dir/de5a8a39-81f2-4538-99d6-6da9bc70e138-util\") pod \"de5a8a39-81f2-4538-99d6-6da9bc70e138\" (UID: \"de5a8a39-81f2-4538-99d6-6da9bc70e138\") "
Oct 31 01:02:14.949104 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:14.948945    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5a8a39-81f2-4538-99d6-6da9bc70e138-bundle\" (UniqueName: \"kubernetes.io/empty-dir/de5a8a39-81f2-4538-99d6-6da9bc70e138-bundle\") pod \"de5a8a39-81f2-4538-99d6-6da9bc70e138\" (UID: \"de5a8a39-81f2-4538-99d6-6da9bc70e138\") "
Oct 31 01:02:14.949383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:14.949359    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5a8a39-81f2-4538-99d6-6da9bc70e138-bundle" pod "de5a8a39-81f2-4538-99d6-6da9bc70e138" (UID: "de5a8a39-81f2-4538-99d6-6da9bc70e138"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:02:14.950919 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:14.950896    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5a8a39-81f2-4538-99d6-6da9bc70e138-kube-api-access-6cpdc" pod "de5a8a39-81f2-4538-99d6-6da9bc70e138" (UID: "de5a8a39-81f2-4538-99d6-6da9bc70e138"). InnerVolumeSpecName "kube-api-access-6cpdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:02:14.953350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:14.953330    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5a8a39-81f2-4538-99d6-6da9bc70e138-util" pod "de5a8a39-81f2-4538-99d6-6da9bc70e138" (UID: "de5a8a39-81f2-4538-99d6-6da9bc70e138"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:02:15.049546 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:15.049522    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/de5a8a39-81f2-4538-99d6-6da9bc70e138-util\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:02:15.049546 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:15.049546    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/de5a8a39-81f2-4538-99d6-6da9bc70e138-bundle\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:02:15.049726 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:15.049556    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cpdc\" (UniqueName: \"kubernetes.io/projected/de5a8a39-81f2-4538-99d6-6da9bc70e138-kube-api-access-6cpdc\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:02:15.173661 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:15.173627    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:15.367256 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:15.367219    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:15.668490 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:15.668420    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:02:15.669214 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:15.669165    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb" event={"ID":"de5a8a39-81f2-4538-99d6-6da9bc70e138","Type":"ContainerDied","Data":"7c6e20bdec16fb44e291831d7385959d3d19807233bd17ae90c71d615a92cfce"}
Oct 31 01:02:15.669300 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:15.669222    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb"
Oct 31 01:02:15.669300 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:15.669240    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c6e20bdec16fb44e291831d7385959d3d19807233bd17ae90c71d615a92cfce"
Oct 31 01:02:16.824474 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:16.824444    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:17.033444 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:17.033388    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:17.691513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:17.691480    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:17.888379 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:17.888344    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:21.657253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:21.657207    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:21.852452 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:21.852395    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:22.674891 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:22.674837    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:22.864257 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:22.864203    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:23.286103 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:23.286060    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:23.471348 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:23.471288    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:23.895706 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:23.895665    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:24.088162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:24.088125    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:28.060539 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:28.060504    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:28.257625 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:28.257582    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:29.179904 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:02:29.179871    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:02:31.558765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:31.558730    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:31.759401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:31.759363    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:32.056044 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.056012    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"]
Oct 31 01:02:32.056299 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.056286    4476 cpu_manager.go:414] "containerMap: removing container" podUID="de5a8a39-81f2-4538-99d6-6da9bc70e138" containerName="extract"
Oct 31 01:02:32.056347 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.056300    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5a8a39-81f2-4538-99d6-6da9bc70e138" containerName="extract"
Oct 31 01:02:32.056347 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.056317    4476 cpu_manager.go:414] "containerMap: removing container" podUID="de5a8a39-81f2-4538-99d6-6da9bc70e138" containerName="pull"
Oct 31 01:02:32.056347 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.056322    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5a8a39-81f2-4538-99d6-6da9bc70e138" containerName="pull"
Oct 31 01:02:32.056347 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.056333    4476 cpu_manager.go:414] "containerMap: removing container" podUID="de5a8a39-81f2-4538-99d6-6da9bc70e138" containerName="util"
Oct 31 01:02:32.056347 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.056340    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5a8a39-81f2-4538-99d6-6da9bc70e138" containerName="util"
Oct 31 01:02:32.056514 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.056381    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5a8a39-81f2-4538-99d6-6da9bc70e138" containerName="extract"
Oct 31 01:02:32.060491 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.060474    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"
Oct 31 01:02:32.164463 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.164439    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 01:02:32.164591 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.164470    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-tkswg\""
Oct 31 01:02:32.164591 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.164485    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 01:02:32.165568 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.165551    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"]
Oct 31 01:02:32.173727 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.173709    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b98861d9-9553-41e4-9c3a-c6c1b672d3af-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl\" (UID: \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"
Oct 31 01:02:32.173783 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.173747    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b98861d9-9553-41e4-9c3a-c6c1b672d3af-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl\" (UID: \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"
Oct 31 01:02:32.173783 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.173777    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kjnd\" (UniqueName: \"kubernetes.io/projected/b98861d9-9553-41e4-9c3a-c6c1b672d3af-kube-api-access-8kjnd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl\" (UID: \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"
Oct 31 01:02:32.274404 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.274373    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b98861d9-9553-41e4-9c3a-c6c1b672d3af-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl\" (UID: \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"
Oct 31 01:02:32.274564 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.274423    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kjnd\" (UniqueName: \"kubernetes.io/projected/b98861d9-9553-41e4-9c3a-c6c1b672d3af-kube-api-access-8kjnd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl\" (UID: \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"
Oct 31 01:02:32.274564 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.274484    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b98861d9-9553-41e4-9c3a-c6c1b672d3af-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl\" (UID: \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"
Oct 31 01:02:32.274752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.274730    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b98861d9-9553-41e4-9c3a-c6c1b672d3af-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl\" (UID: \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"
Oct 31 01:02:32.274821 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.274779    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b98861d9-9553-41e4-9c3a-c6c1b672d3af-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl\" (UID: \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"
Oct 31 01:02:32.331976 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.331914    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kjnd\" (UniqueName: \"kubernetes.io/projected/b98861d9-9553-41e4-9c3a-c6c1b672d3af-kube-api-access-8kjnd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl\" (UID: \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"
Oct 31 01:02:32.372756 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.372730    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"
Oct 31 01:02:32.738906 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:32.738839    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"]
Oct 31 01:02:32.774225 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:02:32.774201    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98861d9_9553_41e4_9c3a_c6c1b672d3af.slice/crio-2064a41d3fcbd935bc450e3e2b6272632973b73a861bce0df0031a8931b44e77 WatchSource:0}: Error finding container 2064a41d3fcbd935bc450e3e2b6272632973b73a861bce0df0031a8931b44e77: Status 404 returned error can't find the container with id 2064a41d3fcbd935bc450e3e2b6272632973b73a861bce0df0031a8931b44e77
Oct 31 01:02:33.733160 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:33.733081    4476 generic.go:356] "Generic (PLEG): container finished" podID="b98861d9-9553-41e4-9c3a-c6c1b672d3af" containerID="2cc29dc8659737ed4871987779ca2f01946f4143d30d47776bf5ecf2d7c900a8" exitCode=0
Oct 31 01:02:33.733160 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:33.733146    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl" event={"ID":"b98861d9-9553-41e4-9c3a-c6c1b672d3af","Type":"ContainerDied","Data":"2cc29dc8659737ed4871987779ca2f01946f4143d30d47776bf5ecf2d7c900a8"}
Oct 31 01:02:33.733493 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:33.733210    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl" event={"ID":"b98861d9-9553-41e4-9c3a-c6c1b672d3af","Type":"ContainerStarted","Data":"2064a41d3fcbd935bc450e3e2b6272632973b73a861bce0df0031a8931b44e77"}
Oct 31 01:02:34.739456 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:34.739415    4476 generic.go:356] "Generic (PLEG): container finished" podID="b98861d9-9553-41e4-9c3a-c6c1b672d3af" containerID="d14a86bbf719be72576385a8476f0e4b3befaae57b1d7e39b6c215e841af00b1" exitCode=0
Oct 31 01:02:34.739836 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:34.739464    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl" event={"ID":"b98861d9-9553-41e4-9c3a-c6c1b672d3af","Type":"ContainerDied","Data":"d14a86bbf719be72576385a8476f0e4b3befaae57b1d7e39b6c215e841af00b1"}
Oct 31 01:02:35.745037 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:35.745001    4476 generic.go:356] "Generic (PLEG): container finished" podID="b98861d9-9553-41e4-9c3a-c6c1b672d3af" containerID="cbb0c0fe031e3b0a69d6fe46dd59b22b5193fa042da59857b4787f82cf53c12c" exitCode=0
Oct 31 01:02:35.745437 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:35.745068    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl" event={"ID":"b98861d9-9553-41e4-9c3a-c6c1b672d3af","Type":"ContainerDied","Data":"cbb0c0fe031e3b0a69d6fe46dd59b22b5193fa042da59857b4787f82cf53c12c"}
Oct 31 01:02:36.270895 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:36.270862    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:36.467281 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:36.467253    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:36.866688 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:36.866667    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"
Oct 31 01:02:36.916781 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:36.916745    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:37.009501 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:37.009437    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98861d9-9553-41e4-9c3a-c6c1b672d3af-kube-api-access-8kjnd\" (UniqueName: \"kubernetes.io/projected/b98861d9-9553-41e4-9c3a-c6c1b672d3af-kube-api-access-8kjnd\") pod \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\" (UID: \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\") "
Oct 31 01:02:37.009625 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:37.009534    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b98861d9-9553-41e4-9c3a-c6c1b672d3af-util\" (UniqueName: \"kubernetes.io/empty-dir/b98861d9-9553-41e4-9c3a-c6c1b672d3af-util\") pod \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\" (UID: \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\") "
Oct 31 01:02:37.009625 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:37.009568    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b98861d9-9553-41e4-9c3a-c6c1b672d3af-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b98861d9-9553-41e4-9c3a-c6c1b672d3af-bundle\") pod \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\" (UID: \"b98861d9-9553-41e4-9c3a-c6c1b672d3af\") "
Oct 31 01:02:37.010002 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:37.009971    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98861d9-9553-41e4-9c3a-c6c1b672d3af-bundle" pod "b98861d9-9553-41e4-9c3a-c6c1b672d3af" (UID: "b98861d9-9553-41e4-9c3a-c6c1b672d3af"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:02:37.011664 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:37.011644    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98861d9-9553-41e4-9c3a-c6c1b672d3af-kube-api-access-8kjnd" pod "b98861d9-9553-41e4-9c3a-c6c1b672d3af" (UID: "b98861d9-9553-41e4-9c3a-c6c1b672d3af"). InnerVolumeSpecName "kube-api-access-8kjnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:02:37.015233 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:37.015184    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98861d9-9553-41e4-9c3a-c6c1b672d3af-util" pod "b98861d9-9553-41e4-9c3a-c6c1b672d3af" (UID: "b98861d9-9553-41e4-9c3a-c6c1b672d3af"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:02:37.110627 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:37.110601    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kjnd\" (UniqueName: \"kubernetes.io/projected/b98861d9-9553-41e4-9c3a-c6c1b672d3af-kube-api-access-8kjnd\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:02:37.110627 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:37.110622    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b98861d9-9553-41e4-9c3a-c6c1b672d3af-util\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:02:37.110627 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:37.110631    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b98861d9-9553-41e4-9c3a-c6c1b672d3af-bundle\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:02:37.111411 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:37.111387    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:37.754157 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:37.754126    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl" event={"ID":"b98861d9-9553-41e4-9c3a-c6c1b672d3af","Type":"ContainerDied","Data":"2064a41d3fcbd935bc450e3e2b6272632973b73a861bce0df0031a8931b44e77"}
Oct 31 01:02:37.754332 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:37.754187    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2064a41d3fcbd935bc450e3e2b6272632973b73a861bce0df0031a8931b44e77"
Oct 31 01:02:37.754332 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:37.754138    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644bq8dl"
Oct 31 01:02:40.829916 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:40.829880    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:41.014592 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:41.014564    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:41.956882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:41.956851    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:42.207982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:42.207869    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:47.094675 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:47.094637    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:47.285972 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:47.285895    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:47.707482 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:47.707444    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:47.891611 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:47.891571    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:48.137264 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:48.137235    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:48.390171 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:48.390057    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:49.077652 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:49.077576    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:53.918695 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:53.918611    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:54.118166 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:54.118125    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:55.483759 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:55.483733    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:55.714071 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:55.714017    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:02:59.180424 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:02:59.180391    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:02:59.901275 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:02:59.901241    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:00.105513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:00.105477    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:00.481391 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:00.481349    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:00.683234 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:00.683163    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:05.286210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:05.286148    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:05.469438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:05.469397    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:10.283977 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:10.283944    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:10.464229 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:10.464156    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:10.691657 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:10.691631    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:10.882633 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:10.882602    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:11.083877 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:11.083841    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:11.273979 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:11.273902    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:14.102107 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:14.102076    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:14.284661 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:14.284632    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:14.628593 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:14.628563    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:16.253796 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.253765    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:16.269343 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.269315    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws"]
Oct 31 01:03:16.269644 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.269628    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b98861d9-9553-41e4-9c3a-c6c1b672d3af" containerName="extract"
Oct 31 01:03:16.269732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.269645    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98861d9-9553-41e4-9c3a-c6c1b672d3af" containerName="extract"
Oct 31 01:03:16.269732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.269669    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b98861d9-9553-41e4-9c3a-c6c1b672d3af" containerName="pull"
Oct 31 01:03:16.269732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.269676    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98861d9-9553-41e4-9c3a-c6c1b672d3af" containerName="pull"
Oct 31 01:03:16.269732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.269704    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b98861d9-9553-41e4-9c3a-c6c1b672d3af" containerName="util"
Oct 31 01:03:16.269732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.269710    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98861d9-9553-41e4-9c3a-c6c1b672d3af" containerName="util"
Oct 31 01:03:16.269963 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.269782    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98861d9-9553-41e4-9c3a-c6c1b672d3af" containerName="extract"
Oct 31 01:03:16.272613 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.272594    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws"
Oct 31 01:03:16.327080 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.327055    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 01:03:16.345071 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.345045    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws"]
Oct 31 01:03:16.375445 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.375421    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g6sh\" (UniqueName: \"kubernetes.io/projected/32e2d8b2-67ec-4f33-abe2-b5e27b1761a9-kube-api-access-6g6sh\") pod \"kube-auth-proxy-7d58c8f9d5-2hkws\" (UID: \"32e2d8b2-67ec-4f33-abe2-b5e27b1761a9\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws"
Oct 31 01:03:16.375564 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.375475    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e2d8b2-67ec-4f33-abe2-b5e27b1761a9-tls-certs\") pod \"kube-auth-proxy-7d58c8f9d5-2hkws\" (UID: \"32e2d8b2-67ec-4f33-abe2-b5e27b1761a9\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws"
Oct 31 01:03:16.375564 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.375538    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32e2d8b2-67ec-4f33-abe2-b5e27b1761a9-tmp\") pod \"kube-auth-proxy-7d58c8f9d5-2hkws\" (UID: \"32e2d8b2-67ec-4f33-abe2-b5e27b1761a9\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws"
Oct 31 01:03:16.417373 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.417348    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 01:03:16.417373 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.417359    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xj6pp\""
Oct 31 01:03:16.466564 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.464744    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:16.476778 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.476746    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e2d8b2-67ec-4f33-abe2-b5e27b1761a9-tls-certs\") pod \"kube-auth-proxy-7d58c8f9d5-2hkws\" (UID: \"32e2d8b2-67ec-4f33-abe2-b5e27b1761a9\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws"
Oct 31 01:03:16.476778 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.476774    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32e2d8b2-67ec-4f33-abe2-b5e27b1761a9-tmp\") pod \"kube-auth-proxy-7d58c8f9d5-2hkws\" (UID: \"32e2d8b2-67ec-4f33-abe2-b5e27b1761a9\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws"
Oct 31 01:03:16.477015 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.476859    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g6sh\" (UniqueName: \"kubernetes.io/projected/32e2d8b2-67ec-4f33-abe2-b5e27b1761a9-kube-api-access-6g6sh\") pod \"kube-auth-proxy-7d58c8f9d5-2hkws\" (UID: \"32e2d8b2-67ec-4f33-abe2-b5e27b1761a9\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws"
Oct 31 01:03:16.479483 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.479460    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e2d8b2-67ec-4f33-abe2-b5e27b1761a9-tls-certs\") pod \"kube-auth-proxy-7d58c8f9d5-2hkws\" (UID: \"32e2d8b2-67ec-4f33-abe2-b5e27b1761a9\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws"
Oct 31 01:03:16.479596 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.479557    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32e2d8b2-67ec-4f33-abe2-b5e27b1761a9-tmp\") pod \"kube-auth-proxy-7d58c8f9d5-2hkws\" (UID: \"32e2d8b2-67ec-4f33-abe2-b5e27b1761a9\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws"
Oct 31 01:03:16.532740 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.532714    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g6sh\" (UniqueName: \"kubernetes.io/projected/32e2d8b2-67ec-4f33-abe2-b5e27b1761a9-kube-api-access-6g6sh\") pod \"kube-auth-proxy-7d58c8f9d5-2hkws\" (UID: \"32e2d8b2-67ec-4f33-abe2-b5e27b1761a9\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws"
Oct 31 01:03:16.583959 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.583936    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws"
Oct 31 01:03:16.651029 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.650973    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:16.956949 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.956927    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws"]
Oct 31 01:03:16.959052 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:03:16.959028    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e2d8b2_67ec_4f33_abe2_b5e27b1761a9.slice/crio-937927e0a5a098fbc620a136a384fbdc67186866ef257c9ef59400ddcaed35e5 WatchSource:0}: Error finding container 937927e0a5a098fbc620a136a384fbdc67186866ef257c9ef59400ddcaed35e5: Status 404 returned error can't find the container with id 937927e0a5a098fbc620a136a384fbdc67186866ef257c9ef59400ddcaed35e5
Oct 31 01:03:16.975791 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:16.975771    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:17.197872 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:17.197843    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:17.603983 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:17.603950    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:17.909670 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:17.909599    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws" event={"ID":"32e2d8b2-67ec-4f33-abe2-b5e27b1761a9","Type":"ContainerStarted","Data":"937927e0a5a098fbc620a136a384fbdc67186866ef257c9ef59400ddcaed35e5"}
Oct 31 01:03:18.718857 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:18.718823    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:19.070327 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.070298    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:19.289711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.289679    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:19.686328 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.686294    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"]
Oct 31 01:03:19.689756 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.689734    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:19.745721 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.745659    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"]
Oct 31 01:03:19.793488 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.793442    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 31 01:03:19.793488 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.793446    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:19.793698 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.793524    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:19.793698 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.793524    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 31 01:03:19.793698 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.793571    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 31 01:03:19.793698 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.793614    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-7zkpb\""
Oct 31 01:03:19.804130 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.804098    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/54f2a8fc-9384-49ec-894f-a92d6152400c-manager-config\") pod \"jobset-controller-manager-54fffd6cd-nfdvt\" (UID: \"54f2a8fc-9384-49ec-894f-a92d6152400c\") " pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:19.804279 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.804153    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnfjf\" (UniqueName: \"kubernetes.io/projected/54f2a8fc-9384-49ec-894f-a92d6152400c-kube-api-access-xnfjf\") pod \"jobset-controller-manager-54fffd6cd-nfdvt\" (UID: \"54f2a8fc-9384-49ec-894f-a92d6152400c\") " pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:19.804279 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.804251    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f2a8fc-9384-49ec-894f-a92d6152400c-metrics-certs\") pod \"jobset-controller-manager-54fffd6cd-nfdvt\" (UID: \"54f2a8fc-9384-49ec-894f-a92d6152400c\") " pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:19.804376 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.804320    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54f2a8fc-9384-49ec-894f-a92d6152400c-cert\") pod \"jobset-controller-manager-54fffd6cd-nfdvt\" (UID: \"54f2a8fc-9384-49ec-894f-a92d6152400c\") " pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:19.905160 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.905122    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/54f2a8fc-9384-49ec-894f-a92d6152400c-manager-config\") pod \"jobset-controller-manager-54fffd6cd-nfdvt\" (UID: \"54f2a8fc-9384-49ec-894f-a92d6152400c\") " pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:19.905327 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.905216    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnfjf\" (UniqueName: \"kubernetes.io/projected/54f2a8fc-9384-49ec-894f-a92d6152400c-kube-api-access-xnfjf\") pod \"jobset-controller-manager-54fffd6cd-nfdvt\" (UID: \"54f2a8fc-9384-49ec-894f-a92d6152400c\") " pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:19.905327 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.905246    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f2a8fc-9384-49ec-894f-a92d6152400c-metrics-certs\") pod \"jobset-controller-manager-54fffd6cd-nfdvt\" (UID: \"54f2a8fc-9384-49ec-894f-a92d6152400c\") " pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:19.905327 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.905285    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54f2a8fc-9384-49ec-894f-a92d6152400c-cert\") pod \"jobset-controller-manager-54fffd6cd-nfdvt\" (UID: \"54f2a8fc-9384-49ec-894f-a92d6152400c\") " pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:19.905799 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.905775    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/54f2a8fc-9384-49ec-894f-a92d6152400c-manager-config\") pod \"jobset-controller-manager-54fffd6cd-nfdvt\" (UID: \"54f2a8fc-9384-49ec-894f-a92d6152400c\") " pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:19.907792 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.907774    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f2a8fc-9384-49ec-894f-a92d6152400c-metrics-certs\") pod \"jobset-controller-manager-54fffd6cd-nfdvt\" (UID: \"54f2a8fc-9384-49ec-894f-a92d6152400c\") " pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:19.907892 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.907875    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54f2a8fc-9384-49ec-894f-a92d6152400c-cert\") pod \"jobset-controller-manager-54fffd6cd-nfdvt\" (UID: \"54f2a8fc-9384-49ec-894f-a92d6152400c\") " pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:19.961074 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:19.961056    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnfjf\" (UniqueName: \"kubernetes.io/projected/54f2a8fc-9384-49ec-894f-a92d6152400c-kube-api-access-xnfjf\") pod \"jobset-controller-manager-54fffd6cd-nfdvt\" (UID: \"54f2a8fc-9384-49ec-894f-a92d6152400c\") " pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:20.003731 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:20.003701    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:20.368431 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:20.368305    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"]
Oct 31 01:03:20.370840 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:03:20.370812    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f2a8fc_9384_49ec_894f_a92d6152400c.slice/crio-bb7b9949b5771720cae1c5fe75b4883eed5a28b090fce9d63069583fc4029f58 WatchSource:0}: Error finding container bb7b9949b5771720cae1c5fe75b4883eed5a28b090fce9d63069583fc4029f58: Status 404 returned error can't find the container with id bb7b9949b5771720cae1c5fe75b4883eed5a28b090fce9d63069583fc4029f58
Oct 31 01:03:20.435721 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:20.435665    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:20.608242 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:20.608210    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:20.923785 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:20.923746    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt" event={"ID":"54f2a8fc-9384-49ec-894f-a92d6152400c","Type":"ContainerStarted","Data":"bb7b9949b5771720cae1c5fe75b4883eed5a28b090fce9d63069583fc4029f58"}
Oct 31 01:03:20.925207 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:20.925163    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws" event={"ID":"32e2d8b2-67ec-4f33-abe2-b5e27b1761a9","Type":"ContainerStarted","Data":"3df81a505db8afee971471a89d625ae315d2be8ae6e0d75be8cf99b98140e372"}
Oct 31 01:03:21.038051 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:21.038005    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-2hkws" podStartSLOduration=2.05767692 podStartE2EDuration="5.037991134s" totalImagesPullingTime="2.980314214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:21.036826194 +0000 UTC m=+410.726709323" watchObservedRunningTime="2026-10-31 01:03:21.037991134 +0000 UTC m=+410.727874259"
Oct 31 01:03:21.903910 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:21.903882    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:22.096055 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:22.096026    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:23.682687 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:23.682622    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:24.942898 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:24.942866    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt" event={"ID":"54f2a8fc-9384-49ec-894f-a92d6152400c","Type":"ContainerStarted","Data":"166911be8985a4f1c5c3b6616e7955bedbeb06822bcdb07c5822e817ade25c17"}
Oct 31 01:03:24.943285 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:24.942971    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:25.056947 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:25.056900    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt" podStartSLOduration=2.290586364 podStartE2EDuration="6.056885741s" totalImagesPullingTime="3.766299377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:25.05556782 +0000 UTC m=+414.745450946" watchObservedRunningTime="2026-10-31 01:03:25.056885741 +0000 UTC m=+414.746768861"
Oct 31 01:03:28.393933 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:28.393904    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:29.179419 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:03:29.179385    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:03:35.952861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:35.952826    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-54fffd6cd-nfdvt"
Oct 31 01:03:51.440645 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:51.440583    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:51.693727 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:51.693648    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:52.361747 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:03:52.361715    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:59.180350 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:03:59.180310    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:04:12.039682 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:12.039648    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:23.118801 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:23.118768    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:23.426904 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:23.426826    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:28.893223 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:28.893192    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:29.179840 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:04:29.179762    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:04:34.036018 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:34.035984    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:34.296544 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:34.296471    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:34.541421 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:34.541387    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:35.009140 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:35.009107    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:38.751111 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:38.751085    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:40.527961 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:40.527928    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:40.711441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:40.711414    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:40.906136 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:40.906043    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:41.097883 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:41.097850    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:41.296610 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:41.296566    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:41.776284 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:41.776212    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:41.983756 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:41.983726    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:42.268540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:42.268507    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:42.465578 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:42.465533    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:42.886431 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:42.886401    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:46.058190 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.058142    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"]
Oct 31 01:04:46.058758 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.058358    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w" podUID="e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" containerName="discovery" containerID="cri-o://e2139180c1d661ddeac3ee3ee6bd7286ca69d093c9c89acbe4745c8e7189cf58" gracePeriod=30
Oct 31 01:04:46.392575 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.392552    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:04:46.449900 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.449874    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-csr-dns-cert\") pod \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") "
Oct 31 01:04:46.450053 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.449909    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-cacerts\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-cacerts\") pod \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") "
Oct 31 01:04:46.450053 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.449955    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-token\" (UniqueName: \"kubernetes.io/projected/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-token\") pod \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") "
Oct 31 01:04:46.450053 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.449991    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-csr-ca-configmap\") pod \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") "
Oct 31 01:04:46.450053 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.450034    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-kubeconfig\") pod \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") "
Oct 31 01:04:46.450284 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.450059    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-kube-api-access-vddkm\" (UniqueName: \"kubernetes.io/projected/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-kube-api-access-vddkm\") pod \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") "
Oct 31 01:04:46.450284 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.450198    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-local-certs\") pod \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\" (UID: \"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef\") "
Oct 31 01:04:46.450893 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.450714    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-csr-ca-configmap" pod "e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" (UID: "e2fd2b99-c816-4a1a-9142-a2ec2d8754ef"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:04:46.452882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.452855    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-kubeconfig" pod "e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" (UID: "e2fd2b99-c816-4a1a-9142-a2ec2d8754ef"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:04:46.452996 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.452886    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-token" pod "e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" (UID: "e2fd2b99-c816-4a1a-9142-a2ec2d8754ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:46.452996 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.452888    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-csr-dns-cert" pod "e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" (UID: "e2fd2b99-c816-4a1a-9142-a2ec2d8754ef"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:04:46.452996 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.452903    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-cacerts" pod "e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" (UID: "e2fd2b99-c816-4a1a-9142-a2ec2d8754ef"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:04:46.452996 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.452918    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-kube-api-access-vddkm" pod "e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" (UID: "e2fd2b99-c816-4a1a-9142-a2ec2d8754ef"). InnerVolumeSpecName "kube-api-access-vddkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:46.454345 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.454325    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-local-certs" pod "e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" (UID: "e2fd2b99-c816-4a1a-9142-a2ec2d8754ef"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:46.551536 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.551502    4476 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-cacerts\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:46.551536 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.551529    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:46.551536 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.551539    4476 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-csr-ca-configmap\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:46.551755 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.551548    4476 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-kubeconfig\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:46.551755 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.551557    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vddkm\" (UniqueName: \"kubernetes.io/projected/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-kube-api-access-vddkm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:46.551755 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.551565    4476 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-local-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:46.551755 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.551573    4476 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef-istio-csr-dns-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:46.793119 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.793086    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"]
Oct 31 01:04:46.793379 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.793367    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" containerName="discovery"
Oct 31 01:04:46.793423 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.793380    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" containerName="discovery"
Oct 31 01:04:46.793465 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.793427    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" containerName="discovery"
Oct 31 01:04:46.796319 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.796305    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:46.853270 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.853242    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 31 01:04:46.853415 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.853250    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 31 01:04:46.853415 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.853276    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szz8g\" (UniqueName: \"kubernetes.io/projected/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-kube-api-access-szz8g\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kzt5t\" (UID: \"9787be82-26ee-4d1c-ad0c-7d2de2f3e065\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:46.853415 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.853383    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kzt5t\" (UID: \"9787be82-26ee-4d1c-ad0c-7d2de2f3e065\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:46.853590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.853425    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 01:04:46.853590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.853442    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kzt5t\" (UID: \"9787be82-26ee-4d1c-ad0c-7d2de2f3e065\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:46.853590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.853555    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 31 01:04:46.853590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.853562    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 01:04:46.853590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.853570    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kzt5t\" (UID: \"9787be82-26ee-4d1c-ad0c-7d2de2f3e065\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:46.853780 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.853618    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-c7bth\""
Oct 31 01:04:46.878231 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.878206    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"]
Oct 31 01:04:46.954563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.954519    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kzt5t\" (UID: \"9787be82-26ee-4d1c-ad0c-7d2de2f3e065\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:46.954735 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.954569    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kzt5t\" (UID: \"9787be82-26ee-4d1c-ad0c-7d2de2f3e065\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:46.954735 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.954612    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szz8g\" (UniqueName: \"kubernetes.io/projected/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-kube-api-access-szz8g\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kzt5t\" (UID: \"9787be82-26ee-4d1c-ad0c-7d2de2f3e065\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:46.954735 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.954650    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kzt5t\" (UID: \"9787be82-26ee-4d1c-ad0c-7d2de2f3e065\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:46.954735 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:04:46.954728    4476 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Oct 31 01:04:46.954931 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:04:46.954807    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-tls-cert podName:9787be82-26ee-4d1c-ad0c-7d2de2f3e065 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:47.45478372 +0000 UTC m=+497.144666856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-kzt5t" (UID: "9787be82-26ee-4d1c-ad0c-7d2de2f3e065") : secret "usage-logs-tenancy-proxy-tls" not found
Oct 31 01:04:46.954931 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:04:46.954732    4476 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 31 01:04:46.954931 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:04:46.954875    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-proxy-source podName:9787be82-26ee-4d1c-ad0c-7d2de2f3e065 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:47.45485844 +0000 UTC m=+497.144741566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-kzt5t" (UID: "9787be82-26ee-4d1c-ad0c-7d2de2f3e065") : configmap "usage-tenancy-proxy-source" not found
Oct 31 01:04:46.955205 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:46.955164    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kzt5t\" (UID: \"9787be82-26ee-4d1c-ad0c-7d2de2f3e065\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:47.011769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.011744    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szz8g\" (UniqueName: \"kubernetes.io/projected/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-kube-api-access-szz8g\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kzt5t\" (UID: \"9787be82-26ee-4d1c-ad0c-7d2de2f3e065\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:47.013309 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.013287    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:47.200614 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.200547    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:47.258140 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.258107    4476 generic.go:356] "Generic (PLEG): container finished" podID="e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" containerID="e2139180c1d661ddeac3ee3ee6bd7286ca69d093c9c89acbe4745c8e7189cf58" exitCode=0
Oct 31 01:04:47.258288 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.258153    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"
Oct 31 01:04:47.258288 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.258151    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w" event={"ID":"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef","Type":"ContainerDied","Data":"e2139180c1d661ddeac3ee3ee6bd7286ca69d093c9c89acbe4745c8e7189cf58"}
Oct 31 01:04:47.258525 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.258493    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w" event={"ID":"e2fd2b99-c816-4a1a-9142-a2ec2d8754ef","Type":"ContainerDied","Data":"34ff2509c60db21056e9e9c83a01d53a44742b744ecb8c90d4110a0e312850ae"}
Oct 31 01:04:47.263618 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.258554    4476 scope.go:122] "RemoveContainer" containerID="e2139180c1d661ddeac3ee3ee6bd7286ca69d093c9c89acbe4745c8e7189cf58"
Oct 31 01:04:47.272098 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.272081    4476 scope.go:122] "RemoveContainer" containerID="e2139180c1d661ddeac3ee3ee6bd7286ca69d093c9c89acbe4745c8e7189cf58"
Oct 31 01:04:47.272336 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:04:47.272318    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2139180c1d661ddeac3ee3ee6bd7286ca69d093c9c89acbe4745c8e7189cf58\": container with ID starting with e2139180c1d661ddeac3ee3ee6bd7286ca69d093c9c89acbe4745c8e7189cf58 not found: ID does not exist" containerID="e2139180c1d661ddeac3ee3ee6bd7286ca69d093c9c89acbe4745c8e7189cf58"
Oct 31 01:04:47.272407 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.272341    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2139180c1d661ddeac3ee3ee6bd7286ca69d093c9c89acbe4745c8e7189cf58"} err="failed to get container status \"e2139180c1d661ddeac3ee3ee6bd7286ca69d093c9c89acbe4745c8e7189cf58\": rpc error: code = NotFound desc = could not find container \"e2139180c1d661ddeac3ee3ee6bd7286ca69d093c9c89acbe4745c8e7189cf58\": container with ID starting with e2139180c1d661ddeac3ee3ee6bd7286ca69d093c9c89acbe4745c8e7189cf58 not found: ID does not exist"
Oct 31 01:04:47.430776 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.430747    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"]
Oct 31 01:04:47.433488 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.433463    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-phf8w"]
Oct 31 01:04:47.458380 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.458324    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kzt5t\" (UID: \"9787be82-26ee-4d1c-ad0c-7d2de2f3e065\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:47.458480 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.458381    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kzt5t\" (UID: \"9787be82-26ee-4d1c-ad0c-7d2de2f3e065\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:47.459118 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.459101    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kzt5t\" (UID: \"9787be82-26ee-4d1c-ad0c-7d2de2f3e065\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:47.462826 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.462803    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9787be82-26ee-4d1c-ad0c-7d2de2f3e065-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kzt5t\" (UID: \"9787be82-26ee-4d1c-ad0c-7d2de2f3e065\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:47.707948 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:47.707920    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:04:48.061282 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:48.061254    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:48.075310 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:48.075285    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"]
Oct 31 01:04:48.078166 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:04:48.078138    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9787be82_26ee_4d1c_ad0c_7d2de2f3e065.slice/crio-09cc412c178432bbefb89768c808ba15ef015184c2329bd6843503dd4d24f613 WatchSource:0}: Error finding container 09cc412c178432bbefb89768c808ba15ef015184c2329bd6843503dd4d24f613: Status 404 returned error can't find the container with id 09cc412c178432bbefb89768c808ba15ef015184c2329bd6843503dd4d24f613
Oct 31 01:04:48.247016 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:48.246993    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:48.263937 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:48.263914    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t" event={"ID":"9787be82-26ee-4d1c-ad0c-7d2de2f3e065","Type":"ContainerStarted","Data":"09cc412c178432bbefb89768c808ba15ef015184c2329bd6843503dd4d24f613"}
Oct 31 01:04:48.446854 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:48.446773    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:48.450778 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:48.450752    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2fd2b99-c816-4a1a-9142-a2ec2d8754ef" path="/var/lib/kubelet/pods/e2fd2b99-c816-4a1a-9142-a2ec2d8754ef/volumes"
Oct 31 01:04:48.686587 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:48.686561    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:48.915291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:48.915264    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:49.095371 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:49.095343    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:49.310880 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:04:49.310854    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:59.182805 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:04:59.182764    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:05:00.045686 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:00.045637    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:01.324123 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:01.324092    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t" event={"ID":"9787be82-26ee-4d1c-ad0c-7d2de2f3e065","Type":"ContainerStarted","Data":"0b336d112c7ac24df819129876416d68057f1073734623e74effbda4db754912"}
Oct 31 01:05:01.324514 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:01.324216    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:05:01.438968 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:01.438905    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t" podStartSLOduration=2.938272458 podStartE2EDuration="15.438886027s" totalImagesPullingTime="12.500613569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:01.43772537 +0000 UTC m=+511.127608500" watchObservedRunningTime="2026-10-31 01:05:01.438886027 +0000 UTC m=+511.128769154"
Oct 31 01:05:12.334002 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:12.333975    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kzt5t"
Oct 31 01:05:16.090245 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:16.090212    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:17.855915 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:17.855880    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:18.193373 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:18.193305    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:18.566691 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:18.566660    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:18.800087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:18.800055    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:19.352358 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.352330    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"]
Oct 31 01:05:19.355549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.355526    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:19.467752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.467719    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 01:05:19.468409 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.468388    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"]
Oct 31 01:05:19.471883 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.471864    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 01:05:19.471883 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.471866    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q2nkq\""
Oct 31 01:05:19.472045 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.471926    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:05:19.472045 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.471926    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:05:19.503906 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.503883    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw6vr\" (UniqueName: \"kubernetes.io/projected/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-kube-api-access-fw6vr\") pod \"maas-api-6f7f744b98-69kfl\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") " pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:19.504016 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.503916    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-openshift-service-ca\") pod \"maas-api-6f7f744b98-69kfl\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") " pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:19.504016 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.503950    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-metrics-certs\") pod \"maas-api-6f7f744b98-69kfl\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") " pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:19.504124 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.504071    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-maas-api-tls\") pod \"maas-api-6f7f744b98-69kfl\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") " pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:19.604820 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.604760    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-openshift-service-ca\") pod \"maas-api-6f7f744b98-69kfl\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") " pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:19.604820 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.604803    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-metrics-certs\") pod \"maas-api-6f7f744b98-69kfl\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") " pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:19.604956 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.604823    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-maas-api-tls\") pod \"maas-api-6f7f744b98-69kfl\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") " pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:19.604956 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.604855    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw6vr\" (UniqueName: \"kubernetes.io/projected/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-kube-api-access-fw6vr\") pod \"maas-api-6f7f744b98-69kfl\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") " pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:19.605504 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.605470    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-openshift-service-ca\") pod \"maas-api-6f7f744b98-69kfl\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") " pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:19.607340 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.607319    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-metrics-certs\") pod \"maas-api-6f7f744b98-69kfl\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") " pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:19.607420 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.607400    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-maas-api-tls\") pod \"maas-api-6f7f744b98-69kfl\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") " pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:19.664347 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.664329    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw6vr\" (UniqueName: \"kubernetes.io/projected/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-kube-api-access-fw6vr\") pod \"maas-api-6f7f744b98-69kfl\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") " pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:19.669187 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:19.669149    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:20.040350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:20.040283    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"]
Oct 31 01:05:20.042986 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:05:20.042960    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1ab368_eaf9_4a2f_8727_2d2097cd9836.slice/crio-89fe1f0ccf20b5934e939f85aa4acb904c46bda7a223a01627995294ef2714e8 WatchSource:0}: Error finding container 89fe1f0ccf20b5934e939f85aa4acb904c46bda7a223a01627995294ef2714e8: Status 404 returned error can't find the container with id 89fe1f0ccf20b5934e939f85aa4acb904c46bda7a223a01627995294ef2714e8
Oct 31 01:05:20.395530 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:20.395449    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl" event={"ID":"ae1ab368-eaf9-4a2f-8727-2d2097cd9836","Type":"ContainerStarted","Data":"89fe1f0ccf20b5934e939f85aa4acb904c46bda7a223a01627995294ef2714e8"}
Oct 31 01:05:27.423776 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:27.423743    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl" event={"ID":"ae1ab368-eaf9-4a2f-8727-2d2097cd9836","Type":"ContainerStarted","Data":"44a81f363d78b54d7c37eb8f4d5c03cffbdfd2cc2db73a0ca4da53b99989a874"}
Oct 31 01:05:27.424161 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:27.423878    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:27.538311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:27.538266    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl" podStartSLOduration=2.062992938 podStartE2EDuration="8.538251242s" totalImagesPullingTime="6.475258304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:27.537052948 +0000 UTC m=+537.226936075" watchObservedRunningTime="2026-10-31 01:05:27.538251242 +0000 UTC m=+537.228134368"
Oct 31 01:05:29.093819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:29.093792    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:29.180459 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:05:29.180423    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:05:33.433693 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:33.433662    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:34.110360 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.110324    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:34.301258 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.301228    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:34.414717 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.414634    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cgcp4"]
Oct 31 01:05:34.723853 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.723754    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-cgcp4"
Oct 31 01:05:34.727956 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.727926    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blsxl\" (UniqueName: \"kubernetes.io/projected/2b14f873-b047-4503-a2c0-724c28db04f8-kube-api-access-blsxl\") pod \"authorino-6f6944bcdd-cgcp4\" (UID: \"2b14f873-b047-4503-a2c0-724c28db04f8\") " pod="kuadrant-system/authorino-6f6944bcdd-cgcp4"
Oct 31 01:05:34.728360 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.728342    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cgcp4"]
Oct 31 01:05:34.728413 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.728375    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-555758456c-mm88j"]
Oct 31 01:05:34.760667 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.760643    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:34.760667 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.760661    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-555758456c-mm88j"]
Oct 31 01:05:34.783075 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.783055    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cfld9\""
Oct 31 01:05:34.788268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.788250    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cgcp4"]
Oct 31 01:05:34.788410 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:05:34.788394    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-blsxl], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-cgcp4" podUID="2b14f873-b047-4503-a2c0-724c28db04f8"
Oct 31 01:05:34.824702 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.824681    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7645cd5f4c-mxk9d"]
Oct 31 01:05:34.828715 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.828693    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blsxl\" (UniqueName: \"kubernetes.io/projected/2b14f873-b047-4503-a2c0-724c28db04f8-kube-api-access-blsxl\") pod \"authorino-6f6944bcdd-cgcp4\" (UID: \"2b14f873-b047-4503-a2c0-724c28db04f8\") " pod="kuadrant-system/authorino-6f6944bcdd-cgcp4"
Oct 31 01:05:34.828822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.828726    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67afcaea-e90e-4616-8dc1-97ecd945c795-maas-api-tls\") pod \"maas-api-555758456c-mm88j\" (UID: \"67afcaea-e90e-4616-8dc1-97ecd945c795\") " pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:34.828822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.828747    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67afcaea-e90e-4616-8dc1-97ecd945c795-metrics-certs\") pod \"maas-api-555758456c-mm88j\" (UID: \"67afcaea-e90e-4616-8dc1-97ecd945c795\") " pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:34.828822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.828763    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwvt6\" (UniqueName: \"kubernetes.io/projected/67afcaea-e90e-4616-8dc1-97ecd945c795-kube-api-access-mwvt6\") pod \"maas-api-555758456c-mm88j\" (UID: \"67afcaea-e90e-4616-8dc1-97ecd945c795\") " pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:34.828822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.828800    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67afcaea-e90e-4616-8dc1-97ecd945c795-openshift-service-ca\") pod \"maas-api-555758456c-mm88j\" (UID: \"67afcaea-e90e-4616-8dc1-97ecd945c795\") " pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:34.829898 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.829881    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 01:05:34.829984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.829921    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 01:05:34.846652 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.846621    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7645cd5f4c-mxk9d"
Oct 31 01:05:34.886721 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.886703    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7645cd5f4c-mxk9d"]
Oct 31 01:05:34.889505 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.889480    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blsxl\" (UniqueName: \"kubernetes.io/projected/2b14f873-b047-4503-a2c0-724c28db04f8-kube-api-access-blsxl\") pod \"authorino-6f6944bcdd-cgcp4\" (UID: \"2b14f873-b047-4503-a2c0-724c28db04f8\") " pod="kuadrant-system/authorino-6f6944bcdd-cgcp4"
Oct 31 01:05:34.929918 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.929894    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67afcaea-e90e-4616-8dc1-97ecd945c795-metrics-certs\") pod \"maas-api-555758456c-mm88j\" (UID: \"67afcaea-e90e-4616-8dc1-97ecd945c795\") " pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:34.930039 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.929921    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwvt6\" (UniqueName: \"kubernetes.io/projected/67afcaea-e90e-4616-8dc1-97ecd945c795-kube-api-access-mwvt6\") pod \"maas-api-555758456c-mm88j\" (UID: \"67afcaea-e90e-4616-8dc1-97ecd945c795\") " pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:34.930039 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.929946    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67afcaea-e90e-4616-8dc1-97ecd945c795-openshift-service-ca\") pod \"maas-api-555758456c-mm88j\" (UID: \"67afcaea-e90e-4616-8dc1-97ecd945c795\") " pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:34.930039 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.929996    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67afcaea-e90e-4616-8dc1-97ecd945c795-maas-api-tls\") pod \"maas-api-555758456c-mm88j\" (UID: \"67afcaea-e90e-4616-8dc1-97ecd945c795\") " pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:34.930039 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.930020    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cxk9\" (UniqueName: \"kubernetes.io/projected/0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae-kube-api-access-7cxk9\") pod \"authorino-7645cd5f4c-mxk9d\" (UID: \"0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae\") " pod="kuadrant-system/authorino-7645cd5f4c-mxk9d"
Oct 31 01:05:34.930714 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.930692    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67afcaea-e90e-4616-8dc1-97ecd945c795-openshift-service-ca\") pod \"maas-api-555758456c-mm88j\" (UID: \"67afcaea-e90e-4616-8dc1-97ecd945c795\") " pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:34.932363 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.932345    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67afcaea-e90e-4616-8dc1-97ecd945c795-metrics-certs\") pod \"maas-api-555758456c-mm88j\" (UID: \"67afcaea-e90e-4616-8dc1-97ecd945c795\") " pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:34.932481 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.932465    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67afcaea-e90e-4616-8dc1-97ecd945c795-maas-api-tls\") pod \"maas-api-555758456c-mm88j\" (UID: \"67afcaea-e90e-4616-8dc1-97ecd945c795\") " pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:34.986541 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:34.986488    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwvt6\" (UniqueName: \"kubernetes.io/projected/67afcaea-e90e-4616-8dc1-97ecd945c795-kube-api-access-mwvt6\") pod \"maas-api-555758456c-mm88j\" (UID: \"67afcaea-e90e-4616-8dc1-97ecd945c795\") " pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:35.030374 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.030350    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cxk9\" (UniqueName: \"kubernetes.io/projected/0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae-kube-api-access-7cxk9\") pod \"authorino-7645cd5f4c-mxk9d\" (UID: \"0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae\") " pod="kuadrant-system/authorino-7645cd5f4c-mxk9d"
Oct 31 01:05:35.087822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.087799    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cxk9\" (UniqueName: \"kubernetes.io/projected/0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae-kube-api-access-7cxk9\") pod \"authorino-7645cd5f4c-mxk9d\" (UID: \"0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae\") " pod="kuadrant-system/authorino-7645cd5f4c-mxk9d"
Oct 31 01:05:35.089604 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.089588    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:35.171491 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.171465    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7645cd5f4c-mxk9d"
Oct 31 01:05:35.205405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.205365    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7645cd5f4c-mxk9d"]
Oct 31 01:05:35.232066 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.232040    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-df49c44f7-nqwtq"]
Oct 31 01:05:35.234690 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.234668    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-df49c44f7-nqwtq"
Oct 31 01:05:35.290191 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.290150    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 01:05:35.290945 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.290924    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-df49c44f7-nqwtq"]
Oct 31 01:05:35.332629 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.332605    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850-tls-cert\") pod \"authorino-df49c44f7-nqwtq\" (UID: \"0fad9b3f-0e9c-48c9-8c38-b904f4d1a850\") " pod="kuadrant-system/authorino-df49c44f7-nqwtq"
Oct 31 01:05:35.332725 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.332644    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6slbh\" (UniqueName: \"kubernetes.io/projected/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850-kube-api-access-6slbh\") pod \"authorino-df49c44f7-nqwtq\" (UID: \"0fad9b3f-0e9c-48c9-8c38-b904f4d1a850\") " pod="kuadrant-system/authorino-df49c44f7-nqwtq"
Oct 31 01:05:35.433806 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.433781    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6slbh\" (UniqueName: \"kubernetes.io/projected/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850-kube-api-access-6slbh\") pod \"authorino-df49c44f7-nqwtq\" (UID: \"0fad9b3f-0e9c-48c9-8c38-b904f4d1a850\") " pod="kuadrant-system/authorino-df49c44f7-nqwtq"
Oct 31 01:05:35.433925 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.433845    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850-tls-cert\") pod \"authorino-df49c44f7-nqwtq\" (UID: \"0fad9b3f-0e9c-48c9-8c38-b904f4d1a850\") " pod="kuadrant-system/authorino-df49c44f7-nqwtq"
Oct 31 01:05:35.436368 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.436349    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850-tls-cert\") pod \"authorino-df49c44f7-nqwtq\" (UID: \"0fad9b3f-0e9c-48c9-8c38-b904f4d1a850\") " pod="kuadrant-system/authorino-df49c44f7-nqwtq"
Oct 31 01:05:35.458381 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.458361    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-cgcp4"
Oct 31 01:05:35.473246 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.473226    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-cgcp4"
Oct 31 01:05:35.493645 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.493626    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6slbh\" (UniqueName: \"kubernetes.io/projected/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850-kube-api-access-6slbh\") pod \"authorino-df49c44f7-nqwtq\" (UID: \"0fad9b3f-0e9c-48c9-8c38-b904f4d1a850\") " pod="kuadrant-system/authorino-df49c44f7-nqwtq"
Oct 31 01:05:35.525676 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.525659    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-555758456c-mm88j"]
Oct 31 01:05:35.527866 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:05:35.527848    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67afcaea_e90e_4616_8dc1_97ecd945c795.slice/crio-9d6bf3e72600d85d5b0d9106b8b631e9f6087be48b4e773e9a0e5a29243b301f WatchSource:0}: Error finding container 9d6bf3e72600d85d5b0d9106b8b631e9f6087be48b4e773e9a0e5a29243b301f: Status 404 returned error can't find the container with id 9d6bf3e72600d85d5b0d9106b8b631e9f6087be48b4e773e9a0e5a29243b301f
Oct 31 01:05:35.534833 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.534814    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b14f873-b047-4503-a2c0-724c28db04f8-kube-api-access-blsxl\" (UniqueName: \"kubernetes.io/projected/2b14f873-b047-4503-a2c0-724c28db04f8-kube-api-access-blsxl\") pod \"2b14f873-b047-4503-a2c0-724c28db04f8\" (UID: \"2b14f873-b047-4503-a2c0-724c28db04f8\") "
Oct 31 01:05:35.536769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.536750    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b14f873-b047-4503-a2c0-724c28db04f8-kube-api-access-blsxl" pod "2b14f873-b047-4503-a2c0-724c28db04f8" (UID: "2b14f873-b047-4503-a2c0-724c28db04f8"). InnerVolumeSpecName "kube-api-access-blsxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:35.553894 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.553874    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-df49c44f7-nqwtq"
Oct 31 01:05:35.557380 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.557362    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7645cd5f4c-mxk9d"]
Oct 31 01:05:35.560710 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:05:35.560689    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ed3ce54_c7db_4fb7_b1d4_237cd0bad9ae.slice/crio-717b15d0a08393fc8412e6950fb152c6baa4b9e7f9a4bfb16614006279412e39 WatchSource:0}: Error finding container 717b15d0a08393fc8412e6950fb152c6baa4b9e7f9a4bfb16614006279412e39: Status 404 returned error can't find the container with id 717b15d0a08393fc8412e6950fb152c6baa4b9e7f9a4bfb16614006279412e39
Oct 31 01:05:35.635650 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.635625    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blsxl\" (UniqueName: \"kubernetes.io/projected/2b14f873-b047-4503-a2c0-724c28db04f8-kube-api-access-blsxl\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:35.919389 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:35.919368    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-df49c44f7-nqwtq"]
Oct 31 01:05:35.921612 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:05:35.921586    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fad9b3f_0e9c_48c9_8c38_b904f4d1a850.slice/crio-992a6dfb99dee238bad45be0c09b1a31f5d75ddf6fd979fc357fe7755d67a0b8 WatchSource:0}: Error finding container 992a6dfb99dee238bad45be0c09b1a31f5d75ddf6fd979fc357fe7755d67a0b8: Status 404 returned error can't find the container with id 992a6dfb99dee238bad45be0c09b1a31f5d75ddf6fd979fc357fe7755d67a0b8
Oct 31 01:05:36.464164 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:36.464137    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j" event={"ID":"67afcaea-e90e-4616-8dc1-97ecd945c795","Type":"ContainerStarted","Data":"0d13c66f4278ea068565ff76bd56b71ad771c71207514212af12c3b46f3faab5"}
Oct 31 01:05:36.464311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:36.464206    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j" event={"ID":"67afcaea-e90e-4616-8dc1-97ecd945c795","Type":"ContainerStarted","Data":"9d6bf3e72600d85d5b0d9106b8b631e9f6087be48b4e773e9a0e5a29243b301f"}
Oct 31 01:05:36.464311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:36.464236    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:36.465298 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:36.465279    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-df49c44f7-nqwtq" event={"ID":"0fad9b3f-0e9c-48c9-8c38-b904f4d1a850","Type":"ContainerStarted","Data":"992a6dfb99dee238bad45be0c09b1a31f5d75ddf6fd979fc357fe7755d67a0b8"}
Oct 31 01:05:36.466453 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:36.466437    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-cgcp4"
Oct 31 01:05:36.466453 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:36.466444    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7645cd5f4c-mxk9d" event={"ID":"0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae","Type":"ContainerStarted","Data":"717b15d0a08393fc8412e6950fb152c6baa4b9e7f9a4bfb16614006279412e39"}
Oct 31 01:05:36.579630 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:36.579553    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j" podStartSLOduration=1.784066994 podStartE2EDuration="2.579541157s" totalImagesPullingTime="795.474163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:36.57847246 +0000 UTC m=+546.268355588" watchObservedRunningTime="2026-10-31 01:05:36.579541157 +0000 UTC m=+546.269424282"
Oct 31 01:05:36.751229 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:36.751199    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cgcp4"]
Oct 31 01:05:36.754975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:36.754952    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cgcp4"]
Oct 31 01:05:38.453442 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:38.453411    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b14f873-b047-4503-a2c0-724c28db04f8" path="/var/lib/kubelet/pods/2b14f873-b047-4503-a2c0-724c28db04f8/volumes"
Oct 31 01:05:39.155297 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:39.155254    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:39.440115 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:39.440037    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:39.642993 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:39.642962    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:40.486264 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:40.486217    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-df49c44f7-nqwtq" event={"ID":"0fad9b3f-0e9c-48c9-8c38-b904f4d1a850","Type":"ContainerStarted","Data":"048ccd0442c4a8f14275e92ccc6805f77a6d97e870b398c66d81ccf32a35f402"}
Oct 31 01:05:40.487994 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:40.487963    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7645cd5f4c-mxk9d" event={"ID":"0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae","Type":"ContainerStarted","Data":"c5a1ac3852876120898d466635fe090b53c96d157a08b4d2402013c5ab9aa3d1"}
Oct 31 01:05:40.487994 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:40.487974    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7645cd5f4c-mxk9d" podUID="0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae" containerName="authorino" containerID="cri-o://c5a1ac3852876120898d466635fe090b53c96d157a08b4d2402013c5ab9aa3d1" gracePeriod=30
Oct 31 01:05:40.600022 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:40.599074    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-df49c44f7-nqwtq" podStartSLOduration=1.6883902050000001 podStartE2EDuration="5.59905664s" totalImagesPullingTime="3.910666435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:40.5983779 +0000 UTC m=+550.288261035" watchObservedRunningTime="2026-10-31 01:05:40.59905664 +0000 UTC m=+550.288939766"
Oct 31 01:05:40.712133 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:40.712075    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7645cd5f4c-mxk9d" podStartSLOduration=2.6052398869999998 podStartE2EDuration="6.712056489s" totalImagesPullingTime="4.106816602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:40.711634305 +0000 UTC m=+550.401517431" watchObservedRunningTime="2026-10-31 01:05:40.712056489 +0000 UTC m=+550.401939617"
Oct 31 01:05:40.759896 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:40.759866    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7645cd5f4c-mxk9d"
Oct 31 01:05:40.787199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:40.785383    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae-kube-api-access-7cxk9\" (UniqueName: \"kubernetes.io/projected/0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae-kube-api-access-7cxk9\") pod \"0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae\" (UID: \"0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae\") "
Oct 31 01:05:40.788598 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:40.788569    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae-kube-api-access-7cxk9" pod "0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae" (UID: "0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae"). InnerVolumeSpecName "kube-api-access-7cxk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:40.886751 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:40.886725    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cxk9\" (UniqueName: \"kubernetes.io/projected/0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae-kube-api-access-7cxk9\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:41.492648 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:41.492614    4476 generic.go:356] "Generic (PLEG): container finished" podID="0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae" containerID="c5a1ac3852876120898d466635fe090b53c96d157a08b4d2402013c5ab9aa3d1" exitCode=0
Oct 31 01:05:41.492834 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:41.492656    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7645cd5f4c-mxk9d"
Oct 31 01:05:41.492834 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:41.492690    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7645cd5f4c-mxk9d" event={"ID":"0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae","Type":"ContainerDied","Data":"c5a1ac3852876120898d466635fe090b53c96d157a08b4d2402013c5ab9aa3d1"}
Oct 31 01:05:41.492834 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:41.492725    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7645cd5f4c-mxk9d" event={"ID":"0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae","Type":"ContainerDied","Data":"717b15d0a08393fc8412e6950fb152c6baa4b9e7f9a4bfb16614006279412e39"}
Oct 31 01:05:41.492834 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:41.492740    4476 scope.go:122] "RemoveContainer" containerID="c5a1ac3852876120898d466635fe090b53c96d157a08b4d2402013c5ab9aa3d1"
Oct 31 01:05:41.502981 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:41.502764    4476 scope.go:122] "RemoveContainer" containerID="c5a1ac3852876120898d466635fe090b53c96d157a08b4d2402013c5ab9aa3d1"
Oct 31 01:05:41.503157 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:05:41.503132    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a1ac3852876120898d466635fe090b53c96d157a08b4d2402013c5ab9aa3d1\": container with ID starting with c5a1ac3852876120898d466635fe090b53c96d157a08b4d2402013c5ab9aa3d1 not found: ID does not exist" containerID="c5a1ac3852876120898d466635fe090b53c96d157a08b4d2402013c5ab9aa3d1"
Oct 31 01:05:41.503286 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:41.503165    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a1ac3852876120898d466635fe090b53c96d157a08b4d2402013c5ab9aa3d1"} err="failed to get container status \"c5a1ac3852876120898d466635fe090b53c96d157a08b4d2402013c5ab9aa3d1\": rpc error: code = NotFound desc = could not find container \"c5a1ac3852876120898d466635fe090b53c96d157a08b4d2402013c5ab9aa3d1\": container with ID starting with c5a1ac3852876120898d466635fe090b53c96d157a08b4d2402013c5ab9aa3d1 not found: ID does not exist"
Oct 31 01:05:41.551504 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:41.551476    4476 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal" podUID="7fc072a2ae8e85ec3464a94d2bac4401" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 31 01:05:41.653984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:41.653950    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7645cd5f4c-mxk9d"]
Oct 31 01:05:41.660039 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:41.660017    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7645cd5f4c-mxk9d"]
Oct 31 01:05:42.451799 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:42.451768    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae" path="/var/lib/kubelet/pods/0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae/volumes"
Oct 31 01:05:42.476049 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:42.476024    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-555758456c-mm88j"
Oct 31 01:05:42.641822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:42.641790    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"]
Oct 31 01:05:42.642006 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:42.641983    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl" podUID="ae1ab368-eaf9-4a2f-8727-2d2097cd9836" containerName="maas-api" containerID="cri-o://44a81f363d78b54d7c37eb8f4d5c03cffbdfd2cc2db73a0ca4da53b99989a874" gracePeriod=30
Oct 31 01:05:42.875331 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:42.875308    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:43.005786 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.005762    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-metrics-certs\") pod \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") "
Oct 31 01:05:43.005923 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.005810    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-openshift-service-ca\") pod \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") "
Oct 31 01:05:43.005923 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.005844    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-maas-api-tls\") pod \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") "
Oct 31 01:05:43.006002 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.005969    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-kube-api-access-fw6vr\" (UniqueName: \"kubernetes.io/projected/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-kube-api-access-fw6vr\") pod \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\" (UID: \"ae1ab368-eaf9-4a2f-8727-2d2097cd9836\") "
Oct 31 01:05:43.006290 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.006263    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-openshift-service-ca" pod "ae1ab368-eaf9-4a2f-8727-2d2097cd9836" (UID: "ae1ab368-eaf9-4a2f-8727-2d2097cd9836"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:43.008242 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.008154    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-maas-api-tls" pod "ae1ab368-eaf9-4a2f-8727-2d2097cd9836" (UID: "ae1ab368-eaf9-4a2f-8727-2d2097cd9836"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:43.008242 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.008209    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-metrics-certs" pod "ae1ab368-eaf9-4a2f-8727-2d2097cd9836" (UID: "ae1ab368-eaf9-4a2f-8727-2d2097cd9836"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:43.008373 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.008282    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-kube-api-access-fw6vr" pod "ae1ab368-eaf9-4a2f-8727-2d2097cd9836" (UID: "ae1ab368-eaf9-4a2f-8727-2d2097cd9836"). InnerVolumeSpecName "kube-api-access-fw6vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:43.107373 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.107336    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw6vr\" (UniqueName: \"kubernetes.io/projected/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-kube-api-access-fw6vr\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:43.107373 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.107372    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:43.107513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.107386    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:43.107513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.107399    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1ab368-eaf9-4a2f-8727-2d2097cd9836-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:43.502429 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.502390    4476 generic.go:356] "Generic (PLEG): container finished" podID="ae1ab368-eaf9-4a2f-8727-2d2097cd9836" containerID="44a81f363d78b54d7c37eb8f4d5c03cffbdfd2cc2db73a0ca4da53b99989a874" exitCode=0
Oct 31 01:05:43.502853 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.502442    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"
Oct 31 01:05:43.502853 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.502446    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl" event={"ID":"ae1ab368-eaf9-4a2f-8727-2d2097cd9836","Type":"ContainerDied","Data":"44a81f363d78b54d7c37eb8f4d5c03cffbdfd2cc2db73a0ca4da53b99989a874"}
Oct 31 01:05:43.502853 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.502497    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl" event={"ID":"ae1ab368-eaf9-4a2f-8727-2d2097cd9836","Type":"ContainerDied","Data":"89fe1f0ccf20b5934e939f85aa4acb904c46bda7a223a01627995294ef2714e8"}
Oct 31 01:05:43.502853 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.502527    4476 scope.go:122] "RemoveContainer" containerID="44a81f363d78b54d7c37eb8f4d5c03cffbdfd2cc2db73a0ca4da53b99989a874"
Oct 31 01:05:43.513391 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.513364    4476 scope.go:122] "RemoveContainer" containerID="44a81f363d78b54d7c37eb8f4d5c03cffbdfd2cc2db73a0ca4da53b99989a874"
Oct 31 01:05:43.513971 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:05:43.513947    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a81f363d78b54d7c37eb8f4d5c03cffbdfd2cc2db73a0ca4da53b99989a874\": container with ID starting with 44a81f363d78b54d7c37eb8f4d5c03cffbdfd2cc2db73a0ca4da53b99989a874 not found: ID does not exist" containerID="44a81f363d78b54d7c37eb8f4d5c03cffbdfd2cc2db73a0ca4da53b99989a874"
Oct 31 01:05:43.514082 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.513977    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a81f363d78b54d7c37eb8f4d5c03cffbdfd2cc2db73a0ca4da53b99989a874"} err="failed to get container status \"44a81f363d78b54d7c37eb8f4d5c03cffbdfd2cc2db73a0ca4da53b99989a874\": rpc error: code = NotFound desc = could not find container \"44a81f363d78b54d7c37eb8f4d5c03cffbdfd2cc2db73a0ca4da53b99989a874\": container with ID starting with 44a81f363d78b54d7c37eb8f4d5c03cffbdfd2cc2db73a0ca4da53b99989a874 not found: ID does not exist"
Oct 31 01:05:43.665743 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.665713    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"]
Oct 31 01:05:43.715232 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:43.715205    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f7f744b98-69kfl"]
Oct 31 01:05:44.452191 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:05:44.452139    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1ab368-eaf9-4a2f-8727-2d2097cd9836" path="/var/lib/kubelet/pods/ae1ab368-eaf9-4a2f-8727-2d2097cd9836/volumes"
Oct 31 01:05:59.180098 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:05:59.180064    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:06:29.179672 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:06:29.179635    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:06:29.410244 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:29.410170    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:06:30.474864 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:30.474836    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:06:30.477424 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:30.477396    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:06:36.065842 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.065809    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-77hxg"]
Oct 31 01:06:36.066253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.066097    4476 cpu_manager.go:414] "containerMap: removing container" podUID="0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae" containerName="authorino"
Oct 31 01:06:36.066253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.066107    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae" containerName="authorino"
Oct 31 01:06:36.066253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.066119    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ae1ab368-eaf9-4a2f-8727-2d2097cd9836" containerName="maas-api"
Oct 31 01:06:36.066253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.066124    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1ab368-eaf9-4a2f-8727-2d2097cd9836" containerName="maas-api"
Oct 31 01:06:36.066253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.066188    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed3ce54-c7db-4fb7-b1d4-237cd0bad9ae" containerName="authorino"
Oct 31 01:06:36.066253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.066202    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1ab368-eaf9-4a2f-8727-2d2097cd9836" containerName="maas-api"
Oct 31 01:06:36.070237 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.070220    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg"
Oct 31 01:06:36.095354 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.095331    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8264d1b-f925-435a-931a-e49a358456c7-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-77hxg\" (UID: \"e8264d1b-f925-435a-931a-e49a358456c7\") " pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg"
Oct 31 01:06:36.095461 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.095369    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wmpn\" (UniqueName: \"kubernetes.io/projected/e8264d1b-f925-435a-931a-e49a358456c7-kube-api-access-2wmpn\") pod \"ai-gateway-controller-7d85675f98-77hxg\" (UID: \"e8264d1b-f925-435a-931a-e49a358456c7\") " pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg"
Oct 31 01:06:36.098930 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.098911    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:06:36.124818 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.124788    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-77hxg"]
Oct 31 01:06:36.172457 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.172438    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 01:06:36.172616 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.172601    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-f6xbl\""
Oct 31 01:06:36.172668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.172621    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 01:06:36.196055 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.196034    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8264d1b-f925-435a-931a-e49a358456c7-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-77hxg\" (UID: \"e8264d1b-f925-435a-931a-e49a358456c7\") " pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg"
Oct 31 01:06:36.196139 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.196068    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wmpn\" (UniqueName: \"kubernetes.io/projected/e8264d1b-f925-435a-931a-e49a358456c7-kube-api-access-2wmpn\") pod \"ai-gateway-controller-7d85675f98-77hxg\" (UID: \"e8264d1b-f925-435a-931a-e49a358456c7\") " pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg"
Oct 31 01:06:36.198865 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.198847    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8264d1b-f925-435a-931a-e49a358456c7-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-77hxg\" (UID: \"e8264d1b-f925-435a-931a-e49a358456c7\") " pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg"
Oct 31 01:06:36.252732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.252708    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wmpn\" (UniqueName: \"kubernetes.io/projected/e8264d1b-f925-435a-931a-e49a358456c7-kube-api-access-2wmpn\") pod \"ai-gateway-controller-7d85675f98-77hxg\" (UID: \"e8264d1b-f925-435a-931a-e49a358456c7\") " pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg"
Oct 31 01:06:36.301449 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.301423    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:06:36.382107 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.382060    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg"
Oct 31 01:06:36.766400 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:36.766203    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-77hxg"]
Oct 31 01:06:36.768793 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:06:36.768767    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8264d1b_f925_435a_931a_e49a358456c7.slice/crio-851f0833dcd07d0d2c93f347ed2d3452924cc17e5f4594843ec73efa35b15238 WatchSource:0}: Error finding container 851f0833dcd07d0d2c93f347ed2d3452924cc17e5f4594843ec73efa35b15238: Status 404 returned error can't find the container with id 851f0833dcd07d0d2c93f347ed2d3452924cc17e5f4594843ec73efa35b15238
Oct 31 01:06:37.707248 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:37.707213    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg" event={"ID":"e8264d1b-f925-435a-931a-e49a358456c7","Type":"ContainerStarted","Data":"851f0833dcd07d0d2c93f347ed2d3452924cc17e5f4594843ec73efa35b15238"}
Oct 31 01:06:39.440598 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:39.440566    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:06:39.716774 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:39.716700    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg" event={"ID":"e8264d1b-f925-435a-931a-e49a358456c7","Type":"ContainerStarted","Data":"9101f3385032fbbd9969f38ae4d1532b6e0fdac95b6c1bc513b00da9131bf2fd"}
Oct 31 01:06:39.716911 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:39.716832    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg"
Oct 31 01:06:39.829694 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:39.829647    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg" podStartSLOduration=1.68526665 podStartE2EDuration="3.829634126s" totalImagesPullingTime="2.144367476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:39.829249593 +0000 UTC m=+609.519132716" watchObservedRunningTime="2026-10-31 01:06:39.829634126 +0000 UTC m=+609.519517252"
Oct 31 01:06:44.093779 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:44.093745    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:06:50.722594 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:50.722566    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg"
Oct 31 01:06:51.074452 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.074423    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p"]
Oct 31 01:06:51.077632 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.077612    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p"
Oct 31 01:06:51.107782 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.107759    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e766f760-98d7-44a0-978e-cf414a9e23b5-metrics-certs\") pod \"ai-gateway-controller-6bb949dc9c-x6t7p\" (UID: \"e766f760-98d7-44a0-978e-cf414a9e23b5\") " pod="opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p"
Oct 31 01:06:51.107873 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.107802    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr46h\" (UniqueName: \"kubernetes.io/projected/e766f760-98d7-44a0-978e-cf414a9e23b5-kube-api-access-vr46h\") pod \"ai-gateway-controller-6bb949dc9c-x6t7p\" (UID: \"e766f760-98d7-44a0-978e-cf414a9e23b5\") " pod="opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p"
Oct 31 01:06:51.136112 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.136089    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p"]
Oct 31 01:06:51.209076 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.209045    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e766f760-98d7-44a0-978e-cf414a9e23b5-metrics-certs\") pod \"ai-gateway-controller-6bb949dc9c-x6t7p\" (UID: \"e766f760-98d7-44a0-978e-cf414a9e23b5\") " pod="opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p"
Oct 31 01:06:51.209217 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.209096    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr46h\" (UniqueName: \"kubernetes.io/projected/e766f760-98d7-44a0-978e-cf414a9e23b5-kube-api-access-vr46h\") pod \"ai-gateway-controller-6bb949dc9c-x6t7p\" (UID: \"e766f760-98d7-44a0-978e-cf414a9e23b5\") " pod="opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p"
Oct 31 01:06:51.211729 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.211700    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e766f760-98d7-44a0-978e-cf414a9e23b5-metrics-certs\") pod \"ai-gateway-controller-6bb949dc9c-x6t7p\" (UID: \"e766f760-98d7-44a0-978e-cf414a9e23b5\") " pod="opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p"
Oct 31 01:06:51.266233 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.266205    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr46h\" (UniqueName: \"kubernetes.io/projected/e766f760-98d7-44a0-978e-cf414a9e23b5-kube-api-access-vr46h\") pod \"ai-gateway-controller-6bb949dc9c-x6t7p\" (UID: \"e766f760-98d7-44a0-978e-cf414a9e23b5\") " pod="opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p"
Oct 31 01:06:51.389382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.389312    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p"
Oct 31 01:06:51.648892 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.648818    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:06:51.753704 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.753674    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p"]
Oct 31 01:06:51.756891 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:06:51.756862    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode766f760_98d7_44a0_978e_cf414a9e23b5.slice/crio-0cfbf8aed842a2c1573525e835e2c66bad7cfd992fee0271a9cb5058681feab7 WatchSource:0}: Error finding container 0cfbf8aed842a2c1573525e835e2c66bad7cfd992fee0271a9cb5058681feab7: Status 404 returned error can't find the container with id 0cfbf8aed842a2c1573525e835e2c66bad7cfd992fee0271a9cb5058681feab7
Oct 31 01:06:51.758154 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.758136    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:06:51.764784 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.764760    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p" event={"ID":"e766f760-98d7-44a0-978e-cf414a9e23b5","Type":"ContainerStarted","Data":"0cfbf8aed842a2c1573525e835e2c66bad7cfd992fee0271a9cb5058681feab7"}
Oct 31 01:06:51.827678 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.827641    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:06:51.949709 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.949650    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv"]
Oct 31 01:06:51.953779 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:51.953765    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv"
Oct 31 01:06:52.007736 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.007699    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 01:06:52.007871 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.007851    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-88657\""
Oct 31 01:06:52.009617 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.009592    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv"]
Oct 31 01:06:52.014274 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.014254    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ab6ca9-406f-471c-a4e4-1cc24dc493cb-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dmjmv\" (UID: \"a7ab6ca9-406f-471c-a4e4-1cc24dc493cb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv"
Oct 31 01:06:52.014377 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.014311    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbzdl\" (UniqueName: \"kubernetes.io/projected/a7ab6ca9-406f-471c-a4e4-1cc24dc493cb-kube-api-access-wbzdl\") pod \"payload-pre-processing-7b98bb678c-dmjmv\" (UID: \"a7ab6ca9-406f-471c-a4e4-1cc24dc493cb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv"
Oct 31 01:06:52.095362 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.095329    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vl9ww"]
Oct 31 01:06:52.098444 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.098424    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-vl9ww"
Oct 31 01:06:52.114701 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.114677    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ab6ca9-406f-471c-a4e4-1cc24dc493cb-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dmjmv\" (UID: \"a7ab6ca9-406f-471c-a4e4-1cc24dc493cb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv"
Oct 31 01:06:52.114800 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.114721    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzdl\" (UniqueName: \"kubernetes.io/projected/a7ab6ca9-406f-471c-a4e4-1cc24dc493cb-kube-api-access-wbzdl\") pod \"payload-pre-processing-7b98bb678c-dmjmv\" (UID: \"a7ab6ca9-406f-471c-a4e4-1cc24dc493cb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv"
Oct 31 01:06:52.115329 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.115310    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ab6ca9-406f-471c-a4e4-1cc24dc493cb-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dmjmv\" (UID: \"a7ab6ca9-406f-471c-a4e4-1cc24dc493cb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv"
Oct 31 01:06:52.154948 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.154928    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vl9ww"]
Oct 31 01:06:52.171352 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.171327    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbzdl\" (UniqueName: \"kubernetes.io/projected/a7ab6ca9-406f-471c-a4e4-1cc24dc493cb-kube-api-access-wbzdl\") pod \"payload-pre-processing-7b98bb678c-dmjmv\" (UID: \"a7ab6ca9-406f-471c-a4e4-1cc24dc493cb\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv"
Oct 31 01:06:52.215340 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.215284    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0881ec8-1f45-459a-b40c-29e2d2246ea0-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-vl9ww\" (UID: \"c0881ec8-1f45-459a-b40c-29e2d2246ea0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vl9ww"
Oct 31 01:06:52.215440 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.215352    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whppn\" (UniqueName: \"kubernetes.io/projected/c0881ec8-1f45-459a-b40c-29e2d2246ea0-kube-api-access-whppn\") pod \"payload-processing-55fbbbf9dc-vl9ww\" (UID: \"c0881ec8-1f45-459a-b40c-29e2d2246ea0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vl9ww"
Oct 31 01:06:52.265361 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.265338    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv"
Oct 31 01:06:52.316056 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.315777    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0881ec8-1f45-459a-b40c-29e2d2246ea0-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-vl9ww\" (UID: \"c0881ec8-1f45-459a-b40c-29e2d2246ea0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vl9ww"
Oct 31 01:06:52.316056 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.315853    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whppn\" (UniqueName: \"kubernetes.io/projected/c0881ec8-1f45-459a-b40c-29e2d2246ea0-kube-api-access-whppn\") pod \"payload-processing-55fbbbf9dc-vl9ww\" (UID: \"c0881ec8-1f45-459a-b40c-29e2d2246ea0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vl9ww"
Oct 31 01:06:52.316707 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.316679    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0881ec8-1f45-459a-b40c-29e2d2246ea0-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-vl9ww\" (UID: \"c0881ec8-1f45-459a-b40c-29e2d2246ea0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vl9ww"
Oct 31 01:06:52.372520 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.372476    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whppn\" (UniqueName: \"kubernetes.io/projected/c0881ec8-1f45-459a-b40c-29e2d2246ea0-kube-api-access-whppn\") pod \"payload-processing-55fbbbf9dc-vl9ww\" (UID: \"c0881ec8-1f45-459a-b40c-29e2d2246ea0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vl9ww"
Oct 31 01:06:52.410241 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.410211    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-vl9ww"
Oct 31 01:06:52.635876 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:06:52.635851    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ab6ca9_406f_471c_a4e4_1cc24dc493cb.slice/crio-9e19338c68808ca0222ca74263c50b325328264989e3a704d90527ea436d8311 WatchSource:0}: Error finding container 9e19338c68808ca0222ca74263c50b325328264989e3a704d90527ea436d8311: Status 404 returned error can't find the container with id 9e19338c68808ca0222ca74263c50b325328264989e3a704d90527ea436d8311
Oct 31 01:06:52.639268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.639040    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv"]
Oct 31 01:06:52.769579 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.769525    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv" event={"ID":"a7ab6ca9-406f-471c-a4e4-1cc24dc493cb","Type":"ContainerStarted","Data":"9e19338c68808ca0222ca74263c50b325328264989e3a704d90527ea436d8311"}
Oct 31 01:06:52.770886 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.770863    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p" event={"ID":"e766f760-98d7-44a0-978e-cf414a9e23b5","Type":"ContainerStarted","Data":"03ddd8ef26b1b3d7bf804ea66f6c76b0ec862874b782b37a85fc7ae96d871004"}
Oct 31 01:06:52.771018 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.771000    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p"
Oct 31 01:06:52.780062 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.780037    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vl9ww"]
Oct 31 01:06:52.883694 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:52.883594    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p" podStartSLOduration=1.034998681 podStartE2EDuration="1.883579394s" totalImagesPullingTime="848.580713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:52.882440226 +0000 UTC m=+622.572323366" watchObservedRunningTime="2026-10-31 01:06:52.883579394 +0000 UTC m=+622.573462519"
Oct 31 01:06:53.778213 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:53.778147    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-vl9ww" event={"ID":"c0881ec8-1f45-459a-b40c-29e2d2246ea0","Type":"ContainerStarted","Data":"bb5995424f25019524f062f5cb846df74add36b7e988f628bb9c1d1d09d026d9"}
Oct 31 01:06:55.788647 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:55.788615    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-vl9ww" event={"ID":"c0881ec8-1f45-459a-b40c-29e2d2246ea0","Type":"ContainerStarted","Data":"6b30c104b6a20367d970f74868d41c69f272a8739d947edca9e33719abc88609"}
Oct 31 01:06:55.789088 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:55.788760    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-vl9ww"
Oct 31 01:06:55.790230 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:55.790196    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv" event={"ID":"a7ab6ca9-406f-471c-a4e4-1cc24dc493cb","Type":"ContainerStarted","Data":"c98a1c92f7bcaeb6c04edb2fa6455b1c2eed8b59ab762ee10759d412d7d7cf9e"}
Oct 31 01:06:55.790373 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:55.790304    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv"
Oct 31 01:06:55.940002 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:55.939958    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-vl9ww" podStartSLOduration=1.241096562 podStartE2EDuration="3.939945721s" totalImagesPullingTime="2.698849159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:55.93902011 +0000 UTC m=+625.628903238" watchObservedRunningTime="2026-10-31 01:06:55.939945721 +0000 UTC m=+625.629828897"
Oct 31 01:06:56.053815 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:06:56.053720    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv" podStartSLOduration=2.205228488 podStartE2EDuration="5.053703633s" totalImagesPullingTime="2.848475145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:56.05276442 +0000 UTC m=+625.742647546" watchObservedRunningTime="2026-10-31 01:06:56.053703633 +0000 UTC m=+625.743586760"
Oct 31 01:06:59.180162 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:06:59.180131    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:07:03.780889 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:03.780862    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6bb949dc9c-x6t7p"
Oct 31 01:07:03.944021 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:03.943992    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-77hxg"]
Oct 31 01:07:03.944224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:03.944199    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg" podUID="e8264d1b-f925-435a-931a-e49a358456c7" containerName="manager" containerID="cri-o://9101f3385032fbbd9969f38ae4d1532b6e0fdac95b6c1bc513b00da9131bf2fd" gracePeriod=30
Oct 31 01:07:04.179611 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.179584    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg"
Oct 31 01:07:04.314941 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.314865    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8264d1b-f925-435a-931a-e49a358456c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8264d1b-f925-435a-931a-e49a358456c7-metrics-certs\") pod \"e8264d1b-f925-435a-931a-e49a358456c7\" (UID: \"e8264d1b-f925-435a-931a-e49a358456c7\") "
Oct 31 01:07:04.314941 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.314900    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8264d1b-f925-435a-931a-e49a358456c7-kube-api-access-2wmpn\" (UniqueName: \"kubernetes.io/projected/e8264d1b-f925-435a-931a-e49a358456c7-kube-api-access-2wmpn\") pod \"e8264d1b-f925-435a-931a-e49a358456c7\" (UID: \"e8264d1b-f925-435a-931a-e49a358456c7\") "
Oct 31 01:07:04.317159 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.317134    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8264d1b-f925-435a-931a-e49a358456c7-metrics-certs" pod "e8264d1b-f925-435a-931a-e49a358456c7" (UID: "e8264d1b-f925-435a-931a-e49a358456c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:07:04.317301 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.317204    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8264d1b-f925-435a-931a-e49a358456c7-kube-api-access-2wmpn" pod "e8264d1b-f925-435a-931a-e49a358456c7" (UID: "e8264d1b-f925-435a-931a-e49a358456c7"). InnerVolumeSpecName "kube-api-access-2wmpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:04.416027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.416000    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8264d1b-f925-435a-931a-e49a358456c7-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:04.416027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.416025    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wmpn\" (UniqueName: \"kubernetes.io/projected/e8264d1b-f925-435a-931a-e49a358456c7-kube-api-access-2wmpn\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:04.827110 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.827073    4476 generic.go:356] "Generic (PLEG): container finished" podID="e8264d1b-f925-435a-931a-e49a358456c7" containerID="9101f3385032fbbd9969f38ae4d1532b6e0fdac95b6c1bc513b00da9131bf2fd" exitCode=0
Oct 31 01:07:04.827540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.827122    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg"
Oct 31 01:07:04.827540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.827122    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg" event={"ID":"e8264d1b-f925-435a-931a-e49a358456c7","Type":"ContainerDied","Data":"9101f3385032fbbd9969f38ae4d1532b6e0fdac95b6c1bc513b00da9131bf2fd"}
Oct 31 01:07:04.827540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.827249    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-77hxg" event={"ID":"e8264d1b-f925-435a-931a-e49a358456c7","Type":"ContainerDied","Data":"851f0833dcd07d0d2c93f347ed2d3452924cc17e5f4594843ec73efa35b15238"}
Oct 31 01:07:04.827540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.827280    4476 scope.go:122] "RemoveContainer" containerID="9101f3385032fbbd9969f38ae4d1532b6e0fdac95b6c1bc513b00da9131bf2fd"
Oct 31 01:07:04.835319 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.835297    4476 scope.go:122] "RemoveContainer" containerID="9101f3385032fbbd9969f38ae4d1532b6e0fdac95b6c1bc513b00da9131bf2fd"
Oct 31 01:07:04.835607 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:07:04.835589    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9101f3385032fbbd9969f38ae4d1532b6e0fdac95b6c1bc513b00da9131bf2fd\": container with ID starting with 9101f3385032fbbd9969f38ae4d1532b6e0fdac95b6c1bc513b00da9131bf2fd not found: ID does not exist" containerID="9101f3385032fbbd9969f38ae4d1532b6e0fdac95b6c1bc513b00da9131bf2fd"
Oct 31 01:07:04.835656 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.835616    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9101f3385032fbbd9969f38ae4d1532b6e0fdac95b6c1bc513b00da9131bf2fd"} err="failed to get container status \"9101f3385032fbbd9969f38ae4d1532b6e0fdac95b6c1bc513b00da9131bf2fd\": rpc error: code = NotFound desc = could not find container \"9101f3385032fbbd9969f38ae4d1532b6e0fdac95b6c1bc513b00da9131bf2fd\": container with ID starting with 9101f3385032fbbd9969f38ae4d1532b6e0fdac95b6c1bc513b00da9131bf2fd not found: ID does not exist"
Oct 31 01:07:04.989149 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:04.989115    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-77hxg"]
Oct 31 01:07:05.038453 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:05.038424    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-77hxg"]
Oct 31 01:07:06.454814 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:06.454768    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8264d1b-f925-435a-931a-e49a358456c7" path="/var/lib/kubelet/pods/e8264d1b-f925-435a-931a-e49a358456c7/volumes"
Oct 31 01:07:06.797586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:06.797555    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dmjmv"
Oct 31 01:07:06.797760 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:06.797617    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-vl9ww"
Oct 31 01:07:11.851453 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:11.851416    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:16.664612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:16.664582    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:22.343623 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:22.343590    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:29.180695 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:07:29.180659    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:07:33.617704 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:33.617669    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:33.805508 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:33.805474    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:39.784606 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:39.784570    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:41.551678 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.551572    4476 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal" podUID="7fc072a2ae8e85ec3464a94d2bac4401" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:07:41.687948 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.687909    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"]
Oct 31 01:07:41.688248 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.688235    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e8264d1b-f925-435a-931a-e49a358456c7" containerName="manager"
Oct 31 01:07:41.688294 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.688250    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8264d1b-f925-435a-931a-e49a358456c7" containerName="manager"
Oct 31 01:07:41.688333 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.688315    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8264d1b-f925-435a-931a-e49a358456c7" containerName="manager"
Oct 31 01:07:41.694945 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.694923    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.722436 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.722400    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b791ac7-2890-4aef-8a04-c977fd48ada1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.722592 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.722447    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b791ac7-2890-4aef-8a04-c977fd48ada1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.722592 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.722526    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zdb9\" (UniqueName: \"kubernetes.io/projected/0b791ac7-2890-4aef-8a04-c977fd48ada1-kube-api-access-2zdb9\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.722680 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.722590    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b791ac7-2890-4aef-8a04-c977fd48ada1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.722680 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.722613    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b791ac7-2890-4aef-8a04-c977fd48ada1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.722753 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.722690    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b791ac7-2890-4aef-8a04-c977fd48ada1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.749592 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.749555    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"]
Oct 31 01:07:41.796650 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.796614    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 01:07:41.796650 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.796642    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q6ptx\""
Oct 31 01:07:41.796650 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.796625    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 01:07:41.796968 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.796694    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 31 01:07:41.823480 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.823398    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b791ac7-2890-4aef-8a04-c977fd48ada1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.823480 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.823444    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zdb9\" (UniqueName: \"kubernetes.io/projected/0b791ac7-2890-4aef-8a04-c977fd48ada1-kube-api-access-2zdb9\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.823480 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.823483    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b791ac7-2890-4aef-8a04-c977fd48ada1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.823737 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.823502    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b791ac7-2890-4aef-8a04-c977fd48ada1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.823737 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.823527    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b791ac7-2890-4aef-8a04-c977fd48ada1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.823737 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.823561    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b791ac7-2890-4aef-8a04-c977fd48ada1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.823897 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.823875    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b791ac7-2890-4aef-8a04-c977fd48ada1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.823936 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.823884    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b791ac7-2890-4aef-8a04-c977fd48ada1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.823972 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.823948    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b791ac7-2890-4aef-8a04-c977fd48ada1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.825557 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.825532    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:41.825870 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.825853    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b791ac7-2890-4aef-8a04-c977fd48ada1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.826228 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.826211    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b791ac7-2890-4aef-8a04-c977fd48ada1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:41.880612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:41.880583    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zdb9\" (UniqueName: \"kubernetes.io/projected/0b791ac7-2890-4aef-8a04-c977fd48ada1-kube-api-access-2zdb9\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8\" (UID: \"0b791ac7-2890-4aef-8a04-c977fd48ada1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:42.007538 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:42.007508    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:42.380380 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:42.380335    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"]
Oct 31 01:07:42.981860 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:42.981830    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8" event={"ID":"0b791ac7-2890-4aef-8a04-c977fd48ada1","Type":"ContainerStarted","Data":"761600638db590bd10d290e176e12aba0701817fa6354554f30a356e73014826"}
Oct 31 01:07:44.990996 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:44.990960    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8" event={"ID":"0b791ac7-2890-4aef-8a04-c977fd48ada1","Type":"ContainerStarted","Data":"fb328872e1eaaf90da99f289b11f07a59de7b83761a03b177edf53bd931a49ce"}
Oct 31 01:07:45.008436 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:45.008397    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:45.105603 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:45.105545    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8" podStartSLOduration=2.185124737 podStartE2EDuration="4.105532106s" totalImagesPullingTime="1.920407369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:45.104992249 +0000 UTC m=+674.794875386" watchObservedRunningTime="2026-10-31 01:07:45.105532106 +0000 UTC m=+674.795415233"
Oct 31 01:07:50.082858 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.082826    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"]
Oct 31 01:07:50.086131 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.086114    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.137665 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.137635    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 31 01:07:50.142909 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.142875    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"]
Oct 31 01:07:50.193203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.193146    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/076bf998-c29d-4915-ba9d-a653b8185e75-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.193203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.193204    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/076bf998-c29d-4915-ba9d-a653b8185e75-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.193432 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.193261    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076bf998-c29d-4915-ba9d-a653b8185e75-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.193432 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.193289    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmd5q\" (UniqueName: \"kubernetes.io/projected/076bf998-c29d-4915-ba9d-a653b8185e75-kube-api-access-pmd5q\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.193432 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.193324    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/076bf998-c29d-4915-ba9d-a653b8185e75-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.193432 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.193355    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/076bf998-c29d-4915-ba9d-a653b8185e75-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.227986 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.227951    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:50.294533 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.294501    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/076bf998-c29d-4915-ba9d-a653b8185e75-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.294533 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.294543    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/076bf998-c29d-4915-ba9d-a653b8185e75-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.294800 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.294562    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/076bf998-c29d-4915-ba9d-a653b8185e75-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.294800 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.294698    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076bf998-c29d-4915-ba9d-a653b8185e75-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.294800 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.294763    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmd5q\" (UniqueName: \"kubernetes.io/projected/076bf998-c29d-4915-ba9d-a653b8185e75-kube-api-access-pmd5q\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.294957 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.294839    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/076bf998-c29d-4915-ba9d-a653b8185e75-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.295018 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.294953    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/076bf998-c29d-4915-ba9d-a653b8185e75-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.295018 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.295004    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076bf998-c29d-4915-ba9d-a653b8185e75-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.295213 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.295168    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/076bf998-c29d-4915-ba9d-a653b8185e75-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.297046 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.297028    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/076bf998-c29d-4915-ba9d-a653b8185e75-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.297334 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.297313    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/076bf998-c29d-4915-ba9d-a653b8185e75-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.353739 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.353667    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmd5q\" (UniqueName: \"kubernetes.io/projected/076bf998-c29d-4915-ba9d-a653b8185e75-kube-api-access-pmd5q\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xnjmn\" (UID: \"076bf998-c29d-4915-ba9d-a653b8185e75\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.398025 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.397982    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:50.772530 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:50.772453    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"]
Oct 31 01:07:50.776480 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:07:50.776453    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076bf998_c29d_4915_ba9d_a653b8185e75.slice/crio-a383ba62119c77854c6db997a5d91eae4710c9d71e2670c970ee00a97dc2d3d9 WatchSource:0}: Error finding container a383ba62119c77854c6db997a5d91eae4710c9d71e2670c970ee00a97dc2d3d9: Status 404 returned error can't find the container with id a383ba62119c77854c6db997a5d91eae4710c9d71e2670c970ee00a97dc2d3d9
Oct 31 01:07:51.016684 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:51.016654    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn" event={"ID":"076bf998-c29d-4915-ba9d-a653b8185e75","Type":"ContainerStarted","Data":"a383ba62119c77854c6db997a5d91eae4710c9d71e2670c970ee00a97dc2d3d9"}
Oct 31 01:07:52.008024 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:52.007992    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:52.020986 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:52.020962    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:52.023321 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:52.023292    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn" event={"ID":"076bf998-c29d-4915-ba9d-a653b8185e75","Type":"ContainerStarted","Data":"0f35c8fe38adca300fe08c36ca9d7b7cd5a0bb3ed3ed29827af0acf94489e7c2"}
Oct 31 01:07:52.034412 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:52.034390    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8"
Oct 31 01:07:52.364036 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:52.363981    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn" podStartSLOduration=1.790122625 podStartE2EDuration="2.363965106s" totalImagesPullingTime="573.842481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:52.363650075 +0000 UTC m=+682.053533201" watchObservedRunningTime="2026-10-31 01:07:52.363965106 +0000 UTC m=+682.053848261"
Oct 31 01:07:52.399001 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:52.398971    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:07:53.231545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.231507    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"]
Oct 31 01:07:53.235557 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.235538    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.291257 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.291222    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"]
Oct 31 01:07:53.335662 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.335634    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0b68e0-1657-45d3-b42f-573f61a52ac2-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.335812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.335676    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0b68e0-1657-45d3-b42f-573f61a52ac2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.335812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.335757    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0b68e0-1657-45d3-b42f-573f61a52ac2-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.335885 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.335827    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nsdj\" (UniqueName: \"kubernetes.io/projected/6f0b68e0-1657-45d3-b42f-573f61a52ac2-kube-api-access-6nsdj\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.335885 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.335853    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0b68e0-1657-45d3-b42f-573f61a52ac2-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.335951 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.335888    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0b68e0-1657-45d3-b42f-573f61a52ac2-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.338073 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.338056    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 31 01:07:53.371031 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.370987    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:53.437094 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.437058    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0b68e0-1657-45d3-b42f-573f61a52ac2-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.437300 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.437101    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0b68e0-1657-45d3-b42f-573f61a52ac2-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.437300 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.437143    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0b68e0-1657-45d3-b42f-573f61a52ac2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.437300 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.437212    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0b68e0-1657-45d3-b42f-573f61a52ac2-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.437300 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.437262    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nsdj\" (UniqueName: \"kubernetes.io/projected/6f0b68e0-1657-45d3-b42f-573f61a52ac2-kube-api-access-6nsdj\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.437523 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.437387    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0b68e0-1657-45d3-b42f-573f61a52ac2-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.437592 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.437538    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0b68e0-1657-45d3-b42f-573f61a52ac2-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.437592 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.437570    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0b68e0-1657-45d3-b42f-573f61a52ac2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.437670 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.437618    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0b68e0-1657-45d3-b42f-573f61a52ac2-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.439684 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.439663    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0b68e0-1657-45d3-b42f-573f61a52ac2-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.439861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.439842    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0b68e0-1657-45d3-b42f-573f61a52ac2-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.494637 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.494569    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nsdj\" (UniqueName: \"kubernetes.io/projected/6f0b68e0-1657-45d3-b42f-573f61a52ac2-kube-api-access-6nsdj\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zclgc\" (UID: \"6f0b68e0-1657-45d3-b42f-573f61a52ac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.569935 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.569905    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:53.939493 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:53.939471    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"]
Oct 31 01:07:53.941353 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:07:53.941321    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0b68e0_1657_45d3_b42f_573f61a52ac2.slice/crio-78c4660692652b82f25319bf6843ab3865808cc0c4db2169a14391cdb1578e2b WatchSource:0}: Error finding container 78c4660692652b82f25319bf6843ab3865808cc0c4db2169a14391cdb1578e2b: Status 404 returned error can't find the container with id 78c4660692652b82f25319bf6843ab3865808cc0c4db2169a14391cdb1578e2b
Oct 31 01:07:54.032695 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:54.032664    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc" event={"ID":"6f0b68e0-1657-45d3-b42f-573f61a52ac2","Type":"ContainerStarted","Data":"78c4660692652b82f25319bf6843ab3865808cc0c4db2169a14391cdb1578e2b"}
Oct 31 01:07:54.431831 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:54.431778    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:55.039103 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:55.039067    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc" event={"ID":"6f0b68e0-1657-45d3-b42f-573f61a52ac2","Type":"ContainerStarted","Data":"e773f69fdfa645a9018d326e4f4e208709c98c214f43c10f171f3bcfa2985dcb"}
Oct 31 01:07:55.152832 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:55.152783    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc" podStartSLOduration=1.5740823019999999 podStartE2EDuration="2.152769746s" totalImagesPullingTime="578.687444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:55.152391184 +0000 UTC m=+684.842274310" watchObservedRunningTime="2026-10-31 01:07:55.152769746 +0000 UTC m=+684.842652875"
Oct 31 01:07:55.570107 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:55.570080    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:07:57.015149 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.015114    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"]
Oct 31 01:07:57.019032 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.019013    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.069679 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.069654    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec32f101-2d93-4344-9497-37200971b9ad-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.069817 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.069692    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec32f101-2d93-4344-9497-37200971b9ad-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.069817 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.069712    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec32f101-2d93-4344-9497-37200971b9ad-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.069899 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.069837    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec32f101-2d93-4344-9497-37200971b9ad-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.069899 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.069885    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf7xc\" (UniqueName: \"kubernetes.io/projected/ec32f101-2d93-4344-9497-37200971b9ad-kube-api-access-sf7xc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.070044 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.069905    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec32f101-2d93-4344-9497-37200971b9ad-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.071337 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.071320    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 31 01:07:57.112381 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.112357    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:57.122072 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.122053    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"]
Oct 31 01:07:57.171318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.171294    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf7xc\" (UniqueName: \"kubernetes.io/projected/ec32f101-2d93-4344-9497-37200971b9ad-kube-api-access-sf7xc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.171454 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.171322    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec32f101-2d93-4344-9497-37200971b9ad-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.171454 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.171359    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec32f101-2d93-4344-9497-37200971b9ad-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.171565 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.171471    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec32f101-2d93-4344-9497-37200971b9ad-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.171565 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.171509    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec32f101-2d93-4344-9497-37200971b9ad-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.171657 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.171569    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec32f101-2d93-4344-9497-37200971b9ad-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.171770 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.171745    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec32f101-2d93-4344-9497-37200971b9ad-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.172022 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.171987    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec32f101-2d93-4344-9497-37200971b9ad-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.172140 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.172041    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec32f101-2d93-4344-9497-37200971b9ad-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.173915 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.173897    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec32f101-2d93-4344-9497-37200971b9ad-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.174052 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.174031    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec32f101-2d93-4344-9497-37200971b9ad-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.228290 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.228262    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf7xc\" (UniqueName: \"kubernetes.io/projected/ec32f101-2d93-4344-9497-37200971b9ad-kube-api-access-sf7xc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w8xgw\" (UID: \"ec32f101-2d93-4344-9497-37200971b9ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.333379 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.333349    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:07:57.701453 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:57.701428    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"]
Oct 31 01:07:57.703611 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:07:57.703585    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec32f101_2d93_4344_9497_37200971b9ad.slice/crio-8afd16ed5c50f889e5c4daac10056f816d31e882e0f703f8f565e7829c499de6 WatchSource:0}: Error finding container 8afd16ed5c50f889e5c4daac10056f816d31e882e0f703f8f565e7829c499de6: Status 404 returned error can't find the container with id 8afd16ed5c50f889e5c4daac10056f816d31e882e0f703f8f565e7829c499de6
Oct 31 01:07:58.052500 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:58.052472    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw" event={"ID":"ec32f101-2d93-4344-9497-37200971b9ad","Type":"ContainerStarted","Data":"8afd16ed5c50f889e5c4daac10056f816d31e882e0f703f8f565e7829c499de6"}
Oct 31 01:07:59.058459 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:59.058426    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw" event={"ID":"ec32f101-2d93-4344-9497-37200971b9ad","Type":"ContainerStarted","Data":"e01bbbeee48b76c8e556e1f6ba7ca4f9b1a46ee93570e49ee0f1112aa5b28e98"}
Oct 31 01:07:59.172396 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:59.172344    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw" podStartSLOduration=2.590982881 podStartE2EDuration="3.172325368s" totalImagesPullingTime="581.342487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:59.171689884 +0000 UTC m=+688.861573013" watchObservedRunningTime="2026-10-31 01:07:59.172325368 +0000 UTC m=+688.862208496"
Oct 31 01:07:59.181060 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:07:59.181025    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:07:59.333952 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:07:59.333869    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:08:00.398840 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:00.398804    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:08:00.412448 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:00.412422    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:08:01.077727 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:01.077699    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xnjmn"
Oct 31 01:08:01.579598 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:01.579565    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:03.570554 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:03.570518    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:08:03.583057 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:03.583024    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:08:04.092884 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:04.092848    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zclgc"
Oct 31 01:08:05.444693 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.444660    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9xsbq"]
Oct 31 01:08:05.448189 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.448160    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.552009 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.551981    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 31 01:08:05.553554 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.553532    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9xsbq"]
Oct 31 01:08:05.562581 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.562558    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982a3abb-abe9-4661-a517-06cc92cb966f-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.562700 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.562588    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982a3abb-abe9-4661-a517-06cc92cb966f-home\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.562700 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.562606    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982a3abb-abe9-4661-a517-06cc92cb966f-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.562700 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.562649    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982a3abb-abe9-4661-a517-06cc92cb966f-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.562818 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.562741    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982a3abb-abe9-4661-a517-06cc92cb966f-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.562818 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.562783    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4pdr\" (UniqueName: \"kubernetes.io/projected/982a3abb-abe9-4661-a517-06cc92cb966f-kube-api-access-b4pdr\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.576531 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.576497    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:05.663902 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.663869    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4pdr\" (UniqueName: \"kubernetes.io/projected/982a3abb-abe9-4661-a517-06cc92cb966f-kube-api-access-b4pdr\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.664074 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.663914    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982a3abb-abe9-4661-a517-06cc92cb966f-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.664074 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.663982    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982a3abb-abe9-4661-a517-06cc92cb966f-home\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.664074 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.664008    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982a3abb-abe9-4661-a517-06cc92cb966f-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.664229 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.664134    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982a3abb-abe9-4661-a517-06cc92cb966f-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.664288 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.664251    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982a3abb-abe9-4661-a517-06cc92cb966f-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.664367 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.664347    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982a3abb-abe9-4661-a517-06cc92cb966f-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.664405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.664375    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982a3abb-abe9-4661-a517-06cc92cb966f-home\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.664443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.664410    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982a3abb-abe9-4661-a517-06cc92cb966f-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.666669 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.666645    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982a3abb-abe9-4661-a517-06cc92cb966f-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.666891 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.666876    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982a3abb-abe9-4661-a517-06cc92cb966f-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.720709 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.720636    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4pdr\" (UniqueName: \"kubernetes.io/projected/982a3abb-abe9-4661-a517-06cc92cb966f-kube-api-access-b4pdr\") pod \"echo-simulator-kserve-64b6f8f96c-9xsbq\" (UID: \"982a3abb-abe9-4661-a517-06cc92cb966f\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:05.761192 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:05.761138    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:06.130401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:06.130376    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9xsbq"]
Oct 31 01:08:06.132833 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:08:06.132809    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982a3abb_abe9_4661_a517_06cc92cb966f.slice/crio-3a7c395b674761c46b44df0c0ec587b909c539c6770c3e044881b28c662839dd WatchSource:0}: Error finding container 3a7c395b674761c46b44df0c0ec587b909c539c6770c3e044881b28c662839dd: Status 404 returned error can't find the container with id 3a7c395b674761c46b44df0c0ec587b909c539c6770c3e044881b28c662839dd
Oct 31 01:08:06.897663 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:06.897634    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:07.098231 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:07.098200    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq" event={"ID":"982a3abb-abe9-4661-a517-06cc92cb966f","Type":"ContainerStarted","Data":"3a7c395b674761c46b44df0c0ec587b909c539c6770c3e044881b28c662839dd"}
Oct 31 01:08:07.333752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:07.333711    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:08:07.351849 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:07.351815    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:08:08.119978 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:08.119938    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w8xgw"
Oct 31 01:08:09.112695 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:09.112649    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq" event={"ID":"982a3abb-abe9-4661-a517-06cc92cb966f","Type":"ContainerStarted","Data":"cf49b5c6baa0f58911ed71c446c387442eb15341a39999881ffad6cb47655dcc"}
Oct 31 01:08:09.228501 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:09.228434    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq" podStartSLOduration=1.853930825 podStartE2EDuration="4.228416923s" totalImagesPullingTime="2.374486098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:09.227383723 +0000 UTC m=+698.917266849" watchObservedRunningTime="2026-10-31 01:08:09.228416923 +0000 UTC m=+698.918300056"
Oct 31 01:08:09.651964 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:09.651901    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:09.761969 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:09.761926    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:13.590837 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.590798    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"]
Oct 31 01:08:13.594533 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.594510    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.646812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.646777    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 01:08:13.646972 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.646928    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x7gsx\""
Oct 31 01:08:13.698235 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.698205    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"]
Oct 31 01:08:13.701952 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.701924    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"]
Oct 31 01:08:13.702115 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.702061    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.741241 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.741209    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b1c85a-f9d0-4502-b529-29df7bc5b837-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.741401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.741265    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.741401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.741316    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.741401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.741334    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.741572 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.741420    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.741572 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.741440    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnz5m\" (UniqueName: \"kubernetes.io/projected/66b1c85a-f9d0-4502-b529-29df7bc5b837-kube-api-access-pnz5m\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.741572 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.741458    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.741572 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.741523    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.741745 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.741591    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.749284 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.749257    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"]
Oct 31 01:08:13.756601 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.756579    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 31 01:08:13.779540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.779486    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:13.843079 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843003    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b1c85a-f9d0-4502-b529-29df7bc5b837-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.843079 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843039    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg4sc\" (UniqueName: \"kubernetes.io/projected/48826212-3a95-460e-9e5e-91ef131c528b-kube-api-access-qg4sc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.843353 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843086    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.843353 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843113    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.843353 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843134    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.843353 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843159    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48826212-3a95-460e-9e5e-91ef131c528b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.843353 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843253    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48826212-3a95-460e-9e5e-91ef131c528b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.843353 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843288    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48826212-3a95-460e-9e5e-91ef131c528b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.843712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843386    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.843712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843418    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnz5m\" (UniqueName: \"kubernetes.io/projected/66b1c85a-f9d0-4502-b529-29df7bc5b837-kube-api-access-pnz5m\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.843712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843447    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.843712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843472    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.843712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843522    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.843712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843565    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48826212-3a95-460e-9e5e-91ef131c528b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.843712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843589    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48826212-3a95-460e-9e5e-91ef131c528b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.843712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843585    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.844243 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843803    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.844243 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.843913    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b1c85a-f9d0-4502-b529-29df7bc5b837-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.844243 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.844061    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.844337 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.844272    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.845790 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.845770    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.846134 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.846094    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.904728 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.904691    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.904958 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.904934    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnz5m\" (UniqueName: \"kubernetes.io/projected/66b1c85a-f9d0-4502-b529-29df7bc5b837-kube-api-access-pnz5m\") pod \"maas-default-gateway-openshift-default-fc8c56547-9ngbl\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.906614 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.906596    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:13.944527 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.944455    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg4sc\" (UniqueName: \"kubernetes.io/projected/48826212-3a95-460e-9e5e-91ef131c528b-kube-api-access-qg4sc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.944527 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.944527    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48826212-3a95-460e-9e5e-91ef131c528b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.944828 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.944546    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48826212-3a95-460e-9e5e-91ef131c528b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.944828 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.944566    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48826212-3a95-460e-9e5e-91ef131c528b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.944828 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.944677    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48826212-3a95-460e-9e5e-91ef131c528b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.944828 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.944697    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48826212-3a95-460e-9e5e-91ef131c528b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.945128 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.945104    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48826212-3a95-460e-9e5e-91ef131c528b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.945436 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.945409    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48826212-3a95-460e-9e5e-91ef131c528b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.945577 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.945559    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48826212-3a95-460e-9e5e-91ef131c528b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.947439 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.947418    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48826212-3a95-460e-9e5e-91ef131c528b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:13.947666 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:13.947649    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48826212-3a95-460e-9e5e-91ef131c528b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:14.000805 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:14.000776    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg4sc\" (UniqueName: \"kubernetes.io/projected/48826212-3a95-460e-9e5e-91ef131c528b-kube-api-access-qg4sc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq\" (UID: \"48826212-3a95-460e-9e5e-91ef131c528b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:14.017111 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:14.017086    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:14.325476 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:14.325448    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"]
Oct 31 01:08:14.327815 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:08:14.327791    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b1c85a_f9d0_4502_b529_29df7bc5b837.slice/crio-49257d2076758c071a3c19d9a1b8deec4ea0822e75fb31cfefda88d2c63f5e1a WatchSource:0}: Error finding container 49257d2076758c071a3c19d9a1b8deec4ea0822e75fb31cfefda88d2c63f5e1a: Status 404 returned error can't find the container with id 49257d2076758c071a3c19d9a1b8deec4ea0822e75fb31cfefda88d2c63f5e1a
Oct 31 01:08:14.385333 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:14.385307    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"]
Oct 31 01:08:14.387387 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:08:14.387350    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48826212_3a95_460e_9e5e_91ef131c528b.slice/crio-37de60a5cc9b8ead6740277ed676eaea1e7cb83b1f6fce09b9a6f77938f33c4e WatchSource:0}: Error finding container 37de60a5cc9b8ead6740277ed676eaea1e7cb83b1f6fce09b9a6f77938f33c4e: Status 404 returned error can't find the container with id 37de60a5cc9b8ead6740277ed676eaea1e7cb83b1f6fce09b9a6f77938f33c4e
Oct 31 01:08:15.141462 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:15.141427    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl" event={"ID":"66b1c85a-f9d0-4502-b529-29df7bc5b837","Type":"ContainerStarted","Data":"49257d2076758c071a3c19d9a1b8deec4ea0822e75fb31cfefda88d2c63f5e1a"}
Oct 31 01:08:15.142938 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:15.142915    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq" event={"ID":"48826212-3a95-460e-9e5e-91ef131c528b","Type":"ContainerStarted","Data":"64df50ac03e467796848ae8f21f017d78a0b7ab24a48bd36a85d690bc066071b"}
Oct 31 01:08:15.143044 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:15.142945    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq" event={"ID":"48826212-3a95-460e-9e5e-91ef131c528b","Type":"ContainerStarted","Data":"37de60a5cc9b8ead6740277ed676eaea1e7cb83b1f6fce09b9a6f77938f33c4e"}
Oct 31 01:08:15.259921 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:15.259818    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq" podStartSLOduration=1.673492971 podStartE2EDuration="2.259804863s" totalImagesPullingTime="586.311892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:15.258244873 +0000 UTC m=+704.948128002" watchObservedRunningTime="2026-10-31 01:08:15.259804863 +0000 UTC m=+704.949687990"
Oct 31 01:08:15.761379 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:15.761326    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:15.773610 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:15.773581    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:16.018269 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:16.018190    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:16.160087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:16.160054    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-9xsbq"
Oct 31 01:08:18.279727 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:18.279688    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:08:18.280137 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:18.279779    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:08:18.280137 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:18.279825    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:08:19.190029 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:19.189994    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl" event={"ID":"66b1c85a-f9d0-4502-b529-29df7bc5b837","Type":"ContainerStarted","Data":"ef22bc96c4c21871dd870b18e9cae8c78a42d4b0676b588c1a9bedf17680931b"}
Oct 31 01:08:19.907261 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:19.907226    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:19.908937 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:19.908911    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-9ngbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 01:08:19.909040 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:19.908962    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl" podUID="66b1c85a-f9d0-4502-b529-29df7bc5b837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 01:08:20.907891 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:20.907856    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-9ngbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 01:08:20.908332 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:20.907908    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl" podUID="66b1c85a-f9d0-4502-b529-29df7bc5b837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 01:08:21.907901 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:21.907858    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-9ngbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 01:08:21.908417 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:21.907931    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl" podUID="66b1c85a-f9d0-4502-b529-29df7bc5b837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 01:08:22.907909 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:22.907863    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-9ngbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 01:08:22.908450 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:22.907941    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl" podUID="66b1c85a-f9d0-4502-b529-29df7bc5b837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 01:08:23.907936 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:23.907895    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-9ngbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 01:08:23.908398 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:23.907967    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl" podUID="66b1c85a-f9d0-4502-b529-29df7bc5b837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 01:08:24.017288 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:24.017253    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:24.030053 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:24.030028    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:24.153483 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:24.153432    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl" podStartSLOduration=7.204587447 podStartE2EDuration="11.153418365s" totalImagesPullingTime="3.948830918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:19.403878887 +0000 UTC m=+709.093762013" watchObservedRunningTime="2026-10-31 01:08:24.153418365 +0000 UTC m=+713.843301489"
Oct 31 01:08:24.223159 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:24.223081    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:08:24.907975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:24.907928    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-9ngbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 01:08:24.908553 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:24.907981    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl" podUID="66b1c85a-f9d0-4502-b529-29df7bc5b837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 01:08:26.530273 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:26.530235    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:26.530785 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:26.530629    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:26.531638 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:26.531603    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:08:29.180115 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:08:29.180079    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:08:34.760067 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:34.760006    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:40.186115 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:40.186083    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:41.328803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:41.328772    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f5785569d-zz4s7"]
Oct 31 01:08:41.336005 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:41.335985    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f5785569d-zz4s7"
Oct 31 01:08:41.390022 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:41.389990    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f5785569d-zz4s7"]
Oct 31 01:08:41.393719 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:41.393683    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14bf16ae-6e30-4983-8f4d-af34c4dca0eb-tls-cert\") pod \"authorino-f5785569d-zz4s7\" (UID: \"14bf16ae-6e30-4983-8f4d-af34c4dca0eb\") " pod="kuadrant-system/authorino-f5785569d-zz4s7"
Oct 31 01:08:41.393887 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:41.393726    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnp7r\" (UniqueName: \"kubernetes.io/projected/14bf16ae-6e30-4983-8f4d-af34c4dca0eb-kube-api-access-wnp7r\") pod \"authorino-f5785569d-zz4s7\" (UID: \"14bf16ae-6e30-4983-8f4d-af34c4dca0eb\") " pod="kuadrant-system/authorino-f5785569d-zz4s7"
Oct 31 01:08:41.494817 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:41.494774    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14bf16ae-6e30-4983-8f4d-af34c4dca0eb-tls-cert\") pod \"authorino-f5785569d-zz4s7\" (UID: \"14bf16ae-6e30-4983-8f4d-af34c4dca0eb\") " pod="kuadrant-system/authorino-f5785569d-zz4s7"
Oct 31 01:08:41.495023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:41.494835    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnp7r\" (UniqueName: \"kubernetes.io/projected/14bf16ae-6e30-4983-8f4d-af34c4dca0eb-kube-api-access-wnp7r\") pod \"authorino-f5785569d-zz4s7\" (UID: \"14bf16ae-6e30-4983-8f4d-af34c4dca0eb\") " pod="kuadrant-system/authorino-f5785569d-zz4s7"
Oct 31 01:08:41.497449 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:41.497427    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14bf16ae-6e30-4983-8f4d-af34c4dca0eb-tls-cert\") pod \"authorino-f5785569d-zz4s7\" (UID: \"14bf16ae-6e30-4983-8f4d-af34c4dca0eb\") " pod="kuadrant-system/authorino-f5785569d-zz4s7"
Oct 31 01:08:41.552320 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:41.552286    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnp7r\" (UniqueName: \"kubernetes.io/projected/14bf16ae-6e30-4983-8f4d-af34c4dca0eb-kube-api-access-wnp7r\") pod \"authorino-f5785569d-zz4s7\" (UID: \"14bf16ae-6e30-4983-8f4d-af34c4dca0eb\") " pod="kuadrant-system/authorino-f5785569d-zz4s7"
Oct 31 01:08:41.647959 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:41.647872    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f5785569d-zz4s7"
Oct 31 01:08:41.996489 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:41.996403    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:42.016170 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:42.016144    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f5785569d-zz4s7"]
Oct 31 01:08:42.017685 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:08:42.017657    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14bf16ae_6e30_4983_8f4d_af34c4dca0eb.slice/crio-3b022798133217b36e67f01aa4b2bc4a33f8e476d5db0e12ee04d3c632a8165a WatchSource:0}: Error finding container 3b022798133217b36e67f01aa4b2bc4a33f8e476d5db0e12ee04d3c632a8165a: Status 404 returned error can't find the container with id 3b022798133217b36e67f01aa4b2bc4a33f8e476d5db0e12ee04d3c632a8165a
Oct 31 01:08:42.189333 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:42.189298    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:42.297996 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:42.297963    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f5785569d-zz4s7" event={"ID":"14bf16ae-6e30-4983-8f4d-af34c4dca0eb","Type":"ContainerStarted","Data":"3b022798133217b36e67f01aa4b2bc4a33f8e476d5db0e12ee04d3c632a8165a"}
Oct 31 01:08:44.309998 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:44.309968    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f5785569d-zz4s7" event={"ID":"14bf16ae-6e30-4983-8f4d-af34c4dca0eb","Type":"ContainerStarted","Data":"77e839d70ecd370f503d1b7fc9a9245fbe4b6c6739466c67cf65179099ba0350"}
Oct 31 01:08:44.447365 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:44.447297    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-f5785569d-zz4s7" podStartSLOduration=2.048451944 podStartE2EDuration="3.447274624s" totalImagesPullingTime="1.39882268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:44.425057413 +0000 UTC m=+734.114940553" watchObservedRunningTime="2026-10-31 01:08:44.447274624 +0000 UTC m=+734.137157752"
Oct 31 01:08:44.477470 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:44.477429    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-df49c44f7-nqwtq"]
Oct 31 01:08:44.477717 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:44.477686    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-df49c44f7-nqwtq" podUID="0fad9b3f-0e9c-48c9-8c38-b904f4d1a850" containerName="authorino" containerID="cri-o://048ccd0442c4a8f14275e92ccc6805f77a6d97e870b398c66d81ccf32a35f402" gracePeriod=30
Oct 31 01:08:44.737661 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:44.737635    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-df49c44f7-nqwtq"
Oct 31 01:08:44.827301 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:44.827266    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850-kube-api-access-6slbh\" (UniqueName: \"kubernetes.io/projected/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850-kube-api-access-6slbh\") pod \"0fad9b3f-0e9c-48c9-8c38-b904f4d1a850\" (UID: \"0fad9b3f-0e9c-48c9-8c38-b904f4d1a850\") "
Oct 31 01:08:44.827495 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:44.827333    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850-tls-cert\" (UniqueName: \"kubernetes.io/secret/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850-tls-cert\") pod \"0fad9b3f-0e9c-48c9-8c38-b904f4d1a850\" (UID: \"0fad9b3f-0e9c-48c9-8c38-b904f4d1a850\") "
Oct 31 01:08:44.829596 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:44.829567    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850-kube-api-access-6slbh" pod "0fad9b3f-0e9c-48c9-8c38-b904f4d1a850" (UID: "0fad9b3f-0e9c-48c9-8c38-b904f4d1a850"). InnerVolumeSpecName "kube-api-access-6slbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:44.840776 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:44.840731    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850-tls-cert" pod "0fad9b3f-0e9c-48c9-8c38-b904f4d1a850" (UID: "0fad9b3f-0e9c-48c9-8c38-b904f4d1a850"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:08:44.928590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:44.928505    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6slbh\" (UniqueName: \"kubernetes.io/projected/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850-kube-api-access-6slbh\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:44.928590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:44.928538    4476 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850-tls-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:45.331198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:45.318601    4476 generic.go:356] "Generic (PLEG): container finished" podID="0fad9b3f-0e9c-48c9-8c38-b904f4d1a850" containerID="048ccd0442c4a8f14275e92ccc6805f77a6d97e870b398c66d81ccf32a35f402" exitCode=0
Oct 31 01:08:45.331198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:45.319372    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-df49c44f7-nqwtq"
Oct 31 01:08:45.331198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:45.319903    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-df49c44f7-nqwtq" event={"ID":"0fad9b3f-0e9c-48c9-8c38-b904f4d1a850","Type":"ContainerDied","Data":"048ccd0442c4a8f14275e92ccc6805f77a6d97e870b398c66d81ccf32a35f402"}
Oct 31 01:08:45.331198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:45.319948    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-df49c44f7-nqwtq" event={"ID":"0fad9b3f-0e9c-48c9-8c38-b904f4d1a850","Type":"ContainerDied","Data":"992a6dfb99dee238bad45be0c09b1a31f5d75ddf6fd979fc357fe7755d67a0b8"}
Oct 31 01:08:45.331198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:45.319969    4476 scope.go:122] "RemoveContainer" containerID="048ccd0442c4a8f14275e92ccc6805f77a6d97e870b398c66d81ccf32a35f402"
Oct 31 01:08:45.347345 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:45.345864    4476 scope.go:122] "RemoveContainer" containerID="048ccd0442c4a8f14275e92ccc6805f77a6d97e870b398c66d81ccf32a35f402"
Oct 31 01:08:45.347345 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:08:45.346296    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048ccd0442c4a8f14275e92ccc6805f77a6d97e870b398c66d81ccf32a35f402\": container with ID starting with 048ccd0442c4a8f14275e92ccc6805f77a6d97e870b398c66d81ccf32a35f402 not found: ID does not exist" containerID="048ccd0442c4a8f14275e92ccc6805f77a6d97e870b398c66d81ccf32a35f402"
Oct 31 01:08:45.347345 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:45.346332    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048ccd0442c4a8f14275e92ccc6805f77a6d97e870b398c66d81ccf32a35f402"} err="failed to get container status \"048ccd0442c4a8f14275e92ccc6805f77a6d97e870b398c66d81ccf32a35f402\": rpc error: code = NotFound desc = could not find container \"048ccd0442c4a8f14275e92ccc6805f77a6d97e870b398c66d81ccf32a35f402\": container with ID starting with 048ccd0442c4a8f14275e92ccc6805f77a6d97e870b398c66d81ccf32a35f402 not found: ID does not exist"
Oct 31 01:08:45.480343 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:45.480308    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-df49c44f7-nqwtq"]
Oct 31 01:08:45.486908 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:45.486880    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-df49c44f7-nqwtq"]
Oct 31 01:08:46.453635 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:46.453601    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fad9b3f-0e9c-48c9-8c38-b904f4d1a850" path="/var/lib/kubelet/pods/0fad9b3f-0e9c-48c9-8c38-b904f4d1a850/volumes"
Oct 31 01:08:53.635269 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:53.635217    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:53.817433 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:53.817394    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:54.752577 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:08:54.752536    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:59.180722 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:08:59.180686    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:09:23.803115 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:23.803067    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:29.180929 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:09:29.180892    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:09:35.099214 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:35.099159    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:40.460011 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:40.459980    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:53.167135 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:53.167102    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:53.368113 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:53.368082    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:53.978832 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:53.978803    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:55.056989 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.056959    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:55.235829 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.235752    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:55.499447 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.499360    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"]
Oct 31 01:09:55.499721 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.499709    4476 cpu_manager.go:414] "containerMap: removing container" podUID="0fad9b3f-0e9c-48c9-8c38-b904f4d1a850" containerName="authorino"
Oct 31 01:09:55.499777 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.499723    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fad9b3f-0e9c-48c9-8c38-b904f4d1a850" containerName="authorino"
Oct 31 01:09:55.499813 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.499800    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fad9b3f-0e9c-48c9-8c38-b904f4d1a850" containerName="authorino"
Oct 31 01:09:55.502627 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.502611    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.554547 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.554517    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-20f0d1b6-openshift-default-dockercfg-vdgdf\""
Oct 31 01:09:55.572222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.572167    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f09923-6594-41c7-aa26-de1b020ad65e-istio-podinfo\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.572408 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.572242    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-credential-socket\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.572408 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.572289    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-istio-data\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.572408 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.572344    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-istio-envoy\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.572408 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.572370    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f09923-6594-41c7-aa26-de1b020ad65e-istio-token\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.572615 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.572433    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-workload-certs\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.572615 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.572465    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnjcr\" (UniqueName: \"kubernetes.io/projected/77f09923-6594-41c7-aa26-de1b020ad65e-kube-api-access-tnjcr\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.572615 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.572500    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-workload-socket\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.572615 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.572602    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f09923-6594-41c7-aa26-de1b020ad65e-istiod-ca-cert\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.607084 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.607047    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"]
Oct 31 01:09:55.630458 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.630412    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:55.673412 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.673381    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f09923-6594-41c7-aa26-de1b020ad65e-istio-podinfo\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.673612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.673417    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-credential-socket\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.673612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.673459    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-istio-data\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.673612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.673483    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-istio-envoy\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.673612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.673512    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f09923-6594-41c7-aa26-de1b020ad65e-istio-token\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.673612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.673537    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-workload-certs\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.673612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.673566    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnjcr\" (UniqueName: \"kubernetes.io/projected/77f09923-6594-41c7-aa26-de1b020ad65e-kube-api-access-tnjcr\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.673612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.673612    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-workload-socket\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.673944 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.673648    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f09923-6594-41c7-aa26-de1b020ad65e-istiod-ca-cert\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.674000 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.673954    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-credential-socket\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.674000 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.673964    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-workload-certs\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.674097 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.674042    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-istio-data\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.674228 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.674206    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-workload-socket\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.674613 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.674592    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f09923-6594-41c7-aa26-de1b020ad65e-istiod-ca-cert\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.675988 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.675970    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-istio-envoy\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.676115 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.676099    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f09923-6594-41c7-aa26-de1b020ad65e-istio-podinfo\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.730509 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.730470    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f09923-6594-41c7-aa26-de1b020ad65e-istio-token\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.730695 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.730678    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnjcr\" (UniqueName: \"kubernetes.io/projected/77f09923-6594-41c7-aa26-de1b020ad65e-kube-api-access-tnjcr\") pod \"e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") " pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.816093 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.816062    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:55.831613 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:55.831554    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:56.238949 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:56.238923    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"]
Oct 31 01:09:56.241398 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:09:56.241367    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f09923_6594_41c7_aa26_de1b020ad65e.slice/crio-fcc43fc6904c696e9c8b1cea1a178058ed36643ca334a94192a97415cbbf2a06 WatchSource:0}: Error finding container fcc43fc6904c696e9c8b1cea1a178058ed36643ca334a94192a97415cbbf2a06: Status 404 returned error can't find the container with id fcc43fc6904c696e9c8b1cea1a178058ed36643ca334a94192a97415cbbf2a06
Oct 31 01:09:56.243367 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:56.243333    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:09:56.243489 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:56.243413    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:09:56.243489 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:56.243461    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:09:56.634352 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:56.634315    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24" event={"ID":"77f09923-6594-41c7-aa26-de1b020ad65e","Type":"ContainerStarted","Data":"8f67e85ea963b57959c0b000a3da71a9cacdf3dbc2bd54cbc53fd545b679f9e4"}
Oct 31 01:09:56.634540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:56.634378    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24" event={"ID":"77f09923-6594-41c7-aa26-de1b020ad65e","Type":"ContainerStarted","Data":"fcc43fc6904c696e9c8b1cea1a178058ed36643ca334a94192a97415cbbf2a06"}
Oct 31 01:09:56.750703 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:56.750632    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24" podStartSLOduration=1.7506170920000002 podStartE2EDuration="1.750617092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:56.749560884 +0000 UTC m=+806.439444016" watchObservedRunningTime="2026-10-31 01:09:56.750617092 +0000 UTC m=+806.440500219"
Oct 31 01:09:56.816920 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:56.816882    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:57.821590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:57.821558    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:58.642907 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:58.642883    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:58.643824 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:58.643806    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:09:59.027289 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:59.027201    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:59.182542 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:09:59.182508    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:09:59.230825 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:09:59.230797    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:00.147864 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.147831    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:00.342295 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.342260    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:00.530769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.530737    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:00.872289 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.872209    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"]
Oct 31 01:10:00.875729 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.875713    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:00.918304 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.918263    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-istio-data\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:00.918680 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.918657    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e817033e-6407-4cea-88a3-ee4b801219b5-istiod-ca-cert\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:00.918955 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.918937    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e817033e-6407-4cea-88a3-ee4b801219b5-istio-podinfo\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:00.919198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.919132    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nffm4\" (UniqueName: \"kubernetes.io/projected/e817033e-6407-4cea-88a3-ee4b801219b5-kube-api-access-nffm4\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:00.919354 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.919220    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-workload-socket\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:00.919354 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.919296    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-credential-socket\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:00.919354 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.919330    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e817033e-6407-4cea-88a3-ee4b801219b5-istio-token\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:00.919737 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.919443    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-workload-certs\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:00.919737 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.919474    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-istio-envoy\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:00.977197 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:00.977140    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"]
Oct 31 01:10:01.009996 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.009960    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:01.021063 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.021026    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-workload-certs\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.021296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.021077    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-istio-envoy\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.021296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.021143    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-istio-data\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.021296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.021167    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e817033e-6407-4cea-88a3-ee4b801219b5-istiod-ca-cert\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.021458 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.021354    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e817033e-6407-4cea-88a3-ee4b801219b5-istio-podinfo\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.021458 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.021415    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nffm4\" (UniqueName: \"kubernetes.io/projected/e817033e-6407-4cea-88a3-ee4b801219b5-kube-api-access-nffm4\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.021458 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.021441    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-workload-socket\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.021623 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.021580    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-workload-certs\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.021724 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.021702    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-credential-socket\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.021826 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.021747    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e817033e-6407-4cea-88a3-ee4b801219b5-istio-token\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.022698 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.022334    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e817033e-6407-4cea-88a3-ee4b801219b5-istiod-ca-cert\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.022698 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.022557    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-istio-data\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.022931 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.022904    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-workload-socket\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.023116 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.023051    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-credential-socket\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.024020 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.023998    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-79f7219a-openshift-default-dockercfg-jwjfd\""
Oct 31 01:10:01.024130 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.024066    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-istio-envoy\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.025657 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.025630    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e817033e-6407-4cea-88a3-ee4b801219b5-istio-podinfo\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.081778 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.081744    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nffm4\" (UniqueName: \"kubernetes.io/projected/e817033e-6407-4cea-88a3-ee4b801219b5-kube-api-access-nffm4\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.081929 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.081826    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e817033e-6407-4cea-88a3-ee4b801219b5-istio-token\") pod \"e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") " pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.189019 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.188940    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:01.240706 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.240667    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:01.614554 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.614529    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"]
Oct 31 01:10:01.615654 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:10:01.615619    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode817033e_6407_4cea_88a3_ee4b801219b5.slice/crio-8b9851685b8e669de804ac3fd6067ad4ef6035df3eaf80cef189955398a9099e WatchSource:0}: Error finding container 8b9851685b8e669de804ac3fd6067ad4ef6035df3eaf80cef189955398a9099e: Status 404 returned error can't find the container with id 8b9851685b8e669de804ac3fd6067ad4ef6035df3eaf80cef189955398a9099e
Oct 31 01:10:01.617689 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.617659    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:10:01.617771 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.617724    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:10:01.617771 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.617753    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:10:01.657686 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:01.657639    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" event={"ID":"e817033e-6407-4cea-88a3-ee4b801219b5","Type":"ContainerStarted","Data":"8b9851685b8e669de804ac3fd6067ad4ef6035df3eaf80cef189955398a9099e"}
Oct 31 01:10:02.143538 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:02.141539    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:02.331650 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:02.331614    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:02.663992 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:02.663958    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" event={"ID":"e817033e-6407-4cea-88a3-ee4b801219b5","Type":"ContainerStarted","Data":"bf7da808748db6cfa121712ce4e7a9850532830c437aa47fc39bfc97e28764e5"}
Oct 31 01:10:02.781190 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:02.781126    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" podStartSLOduration=2.781111961 podStartE2EDuration="2.781111961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:02.779628473 +0000 UTC m=+812.469511613" watchObservedRunningTime="2026-10-31 01:10:02.781111961 +0000 UTC m=+812.470995088"
Oct 31 01:10:03.189359 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:03.189327    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:03.194863 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:03.194839    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:03.411096 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:03.411067    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:03.615295 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:03.615268    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:03.668532 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:03.668505    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:03.669664 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:03.669645    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:04.011662 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:04.011564    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:04.224285 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:04.224253    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:04.412320 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:04.412292    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:04.604539 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:04.604505    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:04.863963 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:04.863934    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:05.071631 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.071588    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:05.117772 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.117694    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"]
Oct 31 01:10:05.121383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.121367    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:05.165023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.164955    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-maas-api-tls\") pod \"maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:05.165193 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.165094    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-metrics-certs\") pod \"maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:05.165250 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.165194    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-openshift-service-ca\") pod \"maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:05.165304 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.165269    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cc54\" (UniqueName: \"kubernetes.io/projected/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-kube-api-access-9cc54\") pod \"maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:05.175223 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.175193    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-20f0d1b6-serving-cert\""
Oct 31 01:10:05.210480 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.210453    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"]
Oct 31 01:10:05.266541 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.266508    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cc54\" (UniqueName: \"kubernetes.io/projected/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-kube-api-access-9cc54\") pod \"maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:05.266711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.266582    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-maas-api-tls\") pod \"maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:05.266711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.266637    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-metrics-certs\") pod \"maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:05.266711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.266694    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-openshift-service-ca\") pod \"maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:05.266878 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:05.266711    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-serving-cert: secret "maas-api-e2e-ait-20f0d1b6-serving-cert" not found
Oct 31 01:10:05.266878 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:05.266774    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-maas-api-tls podName:57bb6d2b-e5fe-4a87-92ee-a5f9fff36602 nodeName:}" failed. No retries permitted until 2026-10-31 01:10:05.766754221 +0000 UTC m=+815.456637326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-maas-api-tls") pod "maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v" (UID: "57bb6d2b-e5fe-4a87-92ee-a5f9fff36602") : secret "maas-api-e2e-ait-20f0d1b6-serving-cert" not found
Oct 31 01:10:05.267487 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.267465    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-openshift-service-ca\") pod \"maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:05.269279 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.269252    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-metrics-certs\") pod \"maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:05.273999 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.273966    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:05.323238 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.323209    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cc54\" (UniqueName: \"kubernetes.io/projected/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-kube-api-access-9cc54\") pod \"maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:05.473871 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.473790    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:05.669935 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.669898    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:05.770488 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.770400    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-maas-api-tls\") pod \"maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:05.772974 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:05.772951    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-maas-api-tls\") pod \"maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:06.034531 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:06.034503    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:06.421848 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:06.421828    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"]
Oct 31 01:10:06.423848 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:10:06.423817    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57bb6d2b_e5fe_4a87_92ee_a5f9fff36602.slice/crio-2b4a68689552159d80605429b0f7f2127050094ae46b9967085e88c4ea1b7b41 WatchSource:0}: Error finding container 2b4a68689552159d80605429b0f7f2127050094ae46b9967085e88c4ea1b7b41: Status 404 returned error can't find the container with id 2b4a68689552159d80605429b0f7f2127050094ae46b9967085e88c4ea1b7b41
Oct 31 01:10:06.682226 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:06.682120    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v" event={"ID":"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602","Type":"ContainerStarted","Data":"2b4a68689552159d80605429b0f7f2127050094ae46b9967085e88c4ea1b7b41"}
Oct 31 01:10:07.688436 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:07.688403    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v" event={"ID":"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602","Type":"ContainerStarted","Data":"b0c89fc553e56cabd79c64aefb2e5ca4d516021b1423ed32afee2a7cd0123406"}
Oct 31 01:10:07.688868 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:07.688662    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:07.803401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:07.803350    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v" podStartSLOduration=1.736544484 podStartE2EDuration="2.803338024s" totalImagesPullingTime="1.06679354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:07.801235316 +0000 UTC m=+817.491118439" watchObservedRunningTime="2026-10-31 01:10:07.803338024 +0000 UTC m=+817.493221150"
Oct 31 01:10:09.392675 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.392645    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:09.575682 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.575648    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:09.749933 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.749857    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"]
Oct 31 01:10:09.753595 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.753578    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"
Oct 31 01:10:09.781090 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.781060    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:09.805489 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.805457    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-79f7219a\""
Oct 31 01:10:09.807846 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.807825    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683400b4-343f-4699-bc65-30e2230105c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf\" (UID: \"683400b4-343f-4699-bc65-30e2230105c1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"
Oct 31 01:10:09.807896 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.807867    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j8jw\" (UniqueName: \"kubernetes.io/projected/683400b4-343f-4699-bc65-30e2230105c1-kube-api-access-5j8jw\") pod \"payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf\" (UID: \"683400b4-343f-4699-bc65-30e2230105c1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"
Oct 31 01:10:09.854688 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.854662    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-79f7219a-dockercfg-lngdk\""
Oct 31 01:10:09.857384 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.857361    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"]
Oct 31 01:10:09.910210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.907978    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"]
Oct 31 01:10:09.915188 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.915121    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"
Oct 31 01:10:09.915816 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.915794    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683400b4-343f-4699-bc65-30e2230105c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf\" (UID: \"683400b4-343f-4699-bc65-30e2230105c1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"
Oct 31 01:10:09.915912 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.915832    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j8jw\" (UniqueName: \"kubernetes.io/projected/683400b4-343f-4699-bc65-30e2230105c1-kube-api-access-5j8jw\") pod \"payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf\" (UID: \"683400b4-343f-4699-bc65-30e2230105c1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"
Oct 31 01:10:09.916565 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.916548    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683400b4-343f-4699-bc65-30e2230105c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf\" (UID: \"683400b4-343f-4699-bc65-30e2230105c1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"
Oct 31 01:10:09.968486 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.968458    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"]
Oct 31 01:10:09.973994 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:09.973959    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j8jw\" (UniqueName: \"kubernetes.io/projected/683400b4-343f-4699-bc65-30e2230105c1-kube-api-access-5j8jw\") pod \"payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf\" (UID: \"683400b4-343f-4699-bc65-30e2230105c1\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"
Oct 31 01:10:10.017236 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.017116    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsnf9\" (UniqueName: \"kubernetes.io/projected/50972db7-acce-465f-899d-048b1662e9e6-kube-api-access-qsnf9\") pod \"payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv\" (UID: \"50972db7-acce-465f-899d-048b1662e9e6\") " pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"
Oct 31 01:10:10.017405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.017266    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50972db7-acce-465f-899d-048b1662e9e6-plugins-config-volume\") pod \"payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv\" (UID: \"50972db7-acce-465f-899d-048b1662e9e6\") " pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"
Oct 31 01:10:10.066458 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.066426    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"
Oct 31 01:10:10.119010 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.118955    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50972db7-acce-465f-899d-048b1662e9e6-plugins-config-volume\") pod \"payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv\" (UID: \"50972db7-acce-465f-899d-048b1662e9e6\") " pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"
Oct 31 01:10:10.119201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.119137    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsnf9\" (UniqueName: \"kubernetes.io/projected/50972db7-acce-465f-899d-048b1662e9e6-kube-api-access-qsnf9\") pod \"payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv\" (UID: \"50972db7-acce-465f-899d-048b1662e9e6\") " pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"
Oct 31 01:10:10.121362 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.121217    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50972db7-acce-465f-899d-048b1662e9e6-plugins-config-volume\") pod \"payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv\" (UID: \"50972db7-acce-465f-899d-048b1662e9e6\") " pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"
Oct 31 01:10:10.176222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.176170    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsnf9\" (UniqueName: \"kubernetes.io/projected/50972db7-acce-465f-899d-048b1662e9e6-kube-api-access-qsnf9\") pod \"payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv\" (UID: \"50972db7-acce-465f-899d-048b1662e9e6\") " pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"
Oct 31 01:10:10.194540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.194508    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:10.228003 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.227976    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"
Oct 31 01:10:10.399574 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.399544    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:10.441350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.441133    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"]
Oct 31 01:10:10.443469 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:10:10.443443    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683400b4_343f_4699_bc65_30e2230105c1.slice/crio-c461c603f1f48272e4e9a0522998a826e3046a01a1f286578dfa264015799ba2 WatchSource:0}: Error finding container c461c603f1f48272e4e9a0522998a826e3046a01a1f286578dfa264015799ba2: Status 404 returned error can't find the container with id c461c603f1f48272e4e9a0522998a826e3046a01a1f286578dfa264015799ba2
Oct 31 01:10:10.603690 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.603662    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"]
Oct 31 01:10:10.605350 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:10:10.605327    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50972db7_acce_465f_899d_048b1662e9e6.slice/crio-d0c308bd7b01e8814a400ac13a8df49e72d5dbd67c6df5c04d8a9d1207434394 WatchSource:0}: Error finding container d0c308bd7b01e8814a400ac13a8df49e72d5dbd67c6df5c04d8a9d1207434394: Status 404 returned error can't find the container with id d0c308bd7b01e8814a400ac13a8df49e72d5dbd67c6df5c04d8a9d1207434394
Oct 31 01:10:10.703474 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.703438    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf" event={"ID":"683400b4-343f-4699-bc65-30e2230105c1","Type":"ContainerStarted","Data":"7f65b4cfdb5be2815af4e2cd9594107302283e5000bf38b3497d3466603dc411"}
Oct 31 01:10:10.703638 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.703593    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"
Oct 31 01:10:10.703685 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.703627    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf" event={"ID":"683400b4-343f-4699-bc65-30e2230105c1","Type":"ContainerStarted","Data":"c461c603f1f48272e4e9a0522998a826e3046a01a1f286578dfa264015799ba2"}
Oct 31 01:10:10.705127 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.705098    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv" event={"ID":"50972db7-acce-465f-899d-048b1662e9e6","Type":"ContainerStarted","Data":"bb245543868c4c1dc86ee80afb43436256be88529b31ae0e02ae73681e4c8429"}
Oct 31 01:10:10.705314 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.705151    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"
Oct 31 01:10:10.705314 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.705214    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv" event={"ID":"50972db7-acce-465f-899d-048b1662e9e6","Type":"ContainerStarted","Data":"d0c308bd7b01e8814a400ac13a8df49e72d5dbd67c6df5c04d8a9d1207434394"}
Oct 31 01:10:10.858770 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.858651    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf" podStartSLOduration=1.858632058 podStartE2EDuration="1.858632058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:10.858393012 +0000 UTC m=+820.548276142" watchObservedRunningTime="2026-10-31 01:10:10.858632058 +0000 UTC m=+820.548515186"
Oct 31 01:10:10.944025 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.943997    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:10.979564 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:10.979499    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv" podStartSLOduration=1.979481352 podStartE2EDuration="1.979481352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:10.977725635 +0000 UTC m=+820.667608765" watchObservedRunningTime="2026-10-31 01:10:10.979481352 +0000 UTC m=+820.669364481"
Oct 31 01:10:12.907440 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:12.907406    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:13.105273 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:13.105241    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:13.698970 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:13.698942    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:15.854033 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:15.853995    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"]
Oct 31 01:10:15.857371 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:15.857356    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:15.909666 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:15.909641    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-79f7219a-serving-cert\""
Oct 31 01:10:15.984311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:15.984264    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"]
Oct 31 01:10:15.984698 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:15.984670    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb82634-defb-47d4-8f9b-eebcc042d6f0-openshift-service-ca\") pod \"maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:15.984797 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:15.984734    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96ch8\" (UniqueName: \"kubernetes.io/projected/3cb82634-defb-47d4-8f9b-eebcc042d6f0-kube-api-access-96ch8\") pod \"maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:15.984797 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:15.984764    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-metrics-certs\") pod \"maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:15.985010 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:15.984986    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-maas-api-tls\") pod \"maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:16.085889 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.085841    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb82634-defb-47d4-8f9b-eebcc042d6f0-openshift-service-ca\") pod \"maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:16.086091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.085918    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96ch8\" (UniqueName: \"kubernetes.io/projected/3cb82634-defb-47d4-8f9b-eebcc042d6f0-kube-api-access-96ch8\") pod \"maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:16.086091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.085949    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-metrics-certs\") pod \"maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:16.086091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.086043    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-maas-api-tls\") pod \"maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:16.086286 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:16.086226    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-serving-cert: secret "maas-api-e2e-derive-79f7219a-serving-cert" not found
Oct 31 01:10:16.086338 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:16.086296    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-maas-api-tls podName:3cb82634-defb-47d4-8f9b-eebcc042d6f0 nodeName:}" failed. No retries permitted until 2026-10-31 01:10:16.586275371 +0000 UTC m=+826.276158478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-maas-api-tls") pod "maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd" (UID: "3cb82634-defb-47d4-8f9b-eebcc042d6f0") : secret "maas-api-e2e-derive-79f7219a-serving-cert" not found
Oct 31 01:10:16.087325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.087300    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb82634-defb-47d4-8f9b-eebcc042d6f0-openshift-service-ca\") pod \"maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:16.090485 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.090455    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-metrics-certs\") pod \"maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:16.094934 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.094912    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:16.143874 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.143795    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96ch8\" (UniqueName: \"kubernetes.io/projected/3cb82634-defb-47d4-8f9b-eebcc042d6f0-kube-api-access-96ch8\") pod \"maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:16.297140 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.297090    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:16.483255 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.482778    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:16.591686 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.591649    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-maas-api-tls\") pod \"maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:16.594724 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.594698    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-maas-api-tls\") pod \"maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:16.684393 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.684357    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:16.794710 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.794675    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:16.951390 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:16.951359    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:17.185276 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:17.179691    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"]
Oct 31 01:10:17.189536 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:10:17.189496    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb82634_defb_47d4_8f9b_eebcc042d6f0.slice/crio-c6bce67fac8dfca830542d6b9dbfb9a80c549f32dc7e370115adabcff6d1a8df WatchSource:0}: Error finding container c6bce67fac8dfca830542d6b9dbfb9a80c549f32dc7e370115adabcff6d1a8df: Status 404 returned error can't find the container with id c6bce67fac8dfca830542d6b9dbfb9a80c549f32dc7e370115adabcff6d1a8df
Oct 31 01:10:17.736893 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:17.736861    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd" event={"ID":"3cb82634-defb-47d4-8f9b-eebcc042d6f0","Type":"ContainerStarted","Data":"c6bce67fac8dfca830542d6b9dbfb9a80c549f32dc7e370115adabcff6d1a8df"}
Oct 31 01:10:18.743778 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:18.743739    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd" event={"ID":"3cb82634-defb-47d4-8f9b-eebcc042d6f0","Type":"ContainerStarted","Data":"cc7022de4b2e8e53e9b72c12d25a0182002c679bb3ca76e36a12c2d9eef2af54"}
Oct 31 01:10:18.744316 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:18.743878    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:18.858306 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:18.858253    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd" podStartSLOduration=2.940185885 podStartE2EDuration="3.85823709s" totalImagesPullingTime="918.051205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:18.856729817 +0000 UTC m=+828.546612968" watchObservedRunningTime="2026-10-31 01:10:18.85823709 +0000 UTC m=+828.548120218"
Oct 31 01:10:19.092061 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:19.092031    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:19.624669 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:19.624628    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"]
Oct 31 01:10:19.628427 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:19.628407    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"
Oct 31 01:10:19.685586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:19.685551    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-935bafe7-dockercfg-6vjb6\""
Oct 31 01:10:19.686841 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:19.686801    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"]
Oct 31 01:10:19.721495 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:19.721459    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qddr\" (UniqueName: \"kubernetes.io/projected/0e78d532-20ba-4881-801a-25d02b170bb0-kube-api-access-9qddr\") pod \"payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l\" (UID: \"0e78d532-20ba-4881-801a-25d02b170bb0\") " pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"
Oct 31 01:10:19.721674 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:19.721524    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78d532-20ba-4881-801a-25d02b170bb0-plugins-config-volume\") pod \"payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l\" (UID: \"0e78d532-20ba-4881-801a-25d02b170bb0\") " pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"
Oct 31 01:10:19.731708 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:19.731689    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-935bafe7\""
Oct 31 01:10:19.822871 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:19.822833    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qddr\" (UniqueName: \"kubernetes.io/projected/0e78d532-20ba-4881-801a-25d02b170bb0-kube-api-access-9qddr\") pod \"payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l\" (UID: \"0e78d532-20ba-4881-801a-25d02b170bb0\") " pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"
Oct 31 01:10:19.823354 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:19.822946    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78d532-20ba-4881-801a-25d02b170bb0-plugins-config-volume\") pod \"payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l\" (UID: \"0e78d532-20ba-4881-801a-25d02b170bb0\") " pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"
Oct 31 01:10:19.823678 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:19.823658    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78d532-20ba-4881-801a-25d02b170bb0-plugins-config-volume\") pod \"payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l\" (UID: \"0e78d532-20ba-4881-801a-25d02b170bb0\") " pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"
Oct 31 01:10:19.885899 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:19.885813    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qddr\" (UniqueName: \"kubernetes.io/projected/0e78d532-20ba-4881-801a-25d02b170bb0-kube-api-access-9qddr\") pod \"payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l\" (UID: \"0e78d532-20ba-4881-801a-25d02b170bb0\") " pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"
Oct 31 01:10:19.941092 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:19.941052    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"
Oct 31 01:10:20.342802 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:20.342770    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"]
Oct 31 01:10:20.345440 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:10:20.345409    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e78d532_20ba_4881_801a_25d02b170bb0.slice/crio-ca3f0ec2bbc4b8f9089125adad0e3be9feca89b23c16c5de018c04f6bdcd9d57 WatchSource:0}: Error finding container ca3f0ec2bbc4b8f9089125adad0e3be9feca89b23c16c5de018c04f6bdcd9d57: Status 404 returned error can't find the container with id ca3f0ec2bbc4b8f9089125adad0e3be9feca89b23c16c5de018c04f6bdcd9d57
Oct 31 01:10:20.582201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:20.578640    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:20.755380 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:20.755292    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l" event={"ID":"0e78d532-20ba-4881-801a-25d02b170bb0","Type":"ContainerStarted","Data":"9b514da15517eb2e024b74b14bcf10e5189a552e06c675e149b6d5012c73afa3"}
Oct 31 01:10:20.755380 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:20.755342    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l" event={"ID":"0e78d532-20ba-4881-801a-25d02b170bb0","Type":"ContainerStarted","Data":"ca3f0ec2bbc4b8f9089125adad0e3be9feca89b23c16c5de018c04f6bdcd9d57"}
Oct 31 01:10:20.755573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:20.755400    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"
Oct 31 01:10:21.478709 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:21.478615    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:21.670692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:21.670642    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:21.712865 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:21.712837    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"
Oct 31 01:10:21.713036 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:21.712910    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"
Oct 31 01:10:21.833214 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:21.833124    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l" podStartSLOduration=2.833107157 podStartE2EDuration="2.833107157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:20.883448548 +0000 UTC m=+830.573331676" watchObservedRunningTime="2026-10-31 01:10:21.833107157 +0000 UTC m=+831.522990286"
Oct 31 01:10:21.866143 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:21.866111    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:24.753941 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:24.753857    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:10:24.789110 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:24.789077    4476 generic.go:356] "Generic (PLEG): container finished" podID="77f09923-6594-41c7-aa26-de1b020ad65e" containerID="8f67e85ea963b57959c0b000a3da71a9cacdf3dbc2bd54cbc53fd545b679f9e4" exitCode=0
Oct 31 01:10:24.789301 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:24.789156    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24" event={"ID":"77f09923-6594-41c7-aa26-de1b020ad65e","Type":"ContainerDied","Data":"8f67e85ea963b57959c0b000a3da71a9cacdf3dbc2bd54cbc53fd545b679f9e4"}
Oct 31 01:10:24.789562 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:24.789548    4476 scope.go:122] "RemoveContainer" containerID="8f67e85ea963b57959c0b000a3da71a9cacdf3dbc2bd54cbc53fd545b679f9e4"
Oct 31 01:10:24.790791 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:24.790751    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:10:24.790900 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:24.790832    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:10:24.790900 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:24.790865    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:10:25.814136 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:25.814105    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24" event={"ID":"77f09923-6594-41c7-aa26-de1b020ad65e","Type":"ContainerStarted","Data":"7c1797d232d6fb5b8723574d14a9efd5df84a05ddd2e188e08b8a2eab6ed6696"}
Oct 31 01:10:25.816507 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:25.816475    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:10:25.817657 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:25.817636    4476 patch_prober.go:29] interesting pod/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 01:10:25.817777 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:25.817678    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24" podUID="77f09923-6594-41c7-aa26-de1b020ad65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 01:10:25.926563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:25.926528    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"]
Oct 31 01:10:25.928975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:25.928953    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:25.958399 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:25.958375    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"]
Oct 31 01:10:25.993987 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:25.993943    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3d4f9b87-serving-cert\""
Oct 31 01:10:26.014903 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.014765    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-openshift-service-ca\") pod \"maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:26.015124 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.014961    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzkwh\" (UniqueName: \"kubernetes.io/projected/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-kube-api-access-xzkwh\") pod \"maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:26.015124 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.015072    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-maas-api-tls\") pod \"maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:26.015276 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.015142    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-metrics-certs\") pod \"maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:26.116307 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.115979    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzkwh\" (UniqueName: \"kubernetes.io/projected/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-kube-api-access-xzkwh\") pod \"maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:26.116307 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.116088    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-maas-api-tls\") pod \"maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:26.116307 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.116136    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-metrics-certs\") pod \"maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:26.118714 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.118687    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-openshift-service-ca\") pod \"maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:26.119076 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:26.119043    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-serving-cert: secret "maas-api-e2e-ait-3d4f9b87-serving-cert" not found
Oct 31 01:10:26.119323 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:26.119307    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-maas-api-tls podName:6b46f41d-5a73-49ed-a5a0-a4c4d041f86b nodeName:}" failed. No retries permitted until 2026-10-31 01:10:26.619251395 +0000 UTC m=+836.309134506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-maas-api-tls") pod "maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg" (UID: "6b46f41d-5a73-49ed-a5a0-a4c4d041f86b") : secret "maas-api-e2e-ait-3d4f9b87-serving-cert" not found
Oct 31 01:10:26.119802 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.116166    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-openshift-service-ca\") pod \"maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:26.123917 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.123893    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-metrics-certs\") pod \"maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:26.210197 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.209813    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzkwh\" (UniqueName: \"kubernetes.io/projected/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-kube-api-access-xzkwh\") pod \"maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:26.251759 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.251727    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:26.485258 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.483028    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:26.624858 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.624814    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-maas-api-tls\") pod \"maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:26.627482 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.627443    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-maas-api-tls\") pod \"maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:26.685201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.684648    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:26.816817 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.816780    4476 patch_prober.go:29] interesting pod/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 01:10:26.817264 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.816855    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24" podUID="77f09923-6594-41c7-aa26-de1b020ad65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 01:10:26.840687 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:26.840660    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:27.277762 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:27.277737    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"]
Oct 31 01:10:27.816718 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:27.816678    4476 patch_prober.go:29] interesting pod/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 01:10:27.816909 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:27.816732    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24" podUID="77f09923-6594-41c7-aa26-de1b020ad65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 01:10:27.824542 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:27.824516    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg" event={"ID":"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b","Type":"ContainerStarted","Data":"f5e253ab3c385f8564660e0e9af32f71a070bb434e3c003d7dd4cffd16019648"}
Oct 31 01:10:28.643421 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:28.643384    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:10:28.816970 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:28.816926    4476 patch_prober.go:29] interesting pod/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 01:10:28.817435 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:28.817001    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24" podUID="77f09923-6594-41c7-aa26-de1b020ad65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 01:10:28.832371 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:28.832342    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg" event={"ID":"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b","Type":"ContainerStarted","Data":"5cc2f875d1e7481c1532c1dc0ede758396b81871a588da047bd511124139f5ce"}
Oct 31 01:10:28.832509 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:28.832449    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:29.005258 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:29.004167    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg" podStartSLOduration=3.179770114 podStartE2EDuration="4.004147049s" totalImagesPullingTime="824.376935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:28.997597663 +0000 UTC m=+838.687480791" watchObservedRunningTime="2026-10-31 01:10:29.004147049 +0000 UTC m=+838.694030176"
Oct 31 01:10:29.231885 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:29.231852    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:10:29.817186 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:29.817120    4476 patch_prober.go:29] interesting pod/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 01:10:29.817673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:29.817209    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24" podUID="77f09923-6594-41c7-aa26-de1b020ad65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 01:10:30.817368 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:30.817329    4476 patch_prober.go:29] interesting pod/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 31 01:10:30.817859 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:30.817392    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24" podUID="77f09923-6594-41c7-aa26-de1b020ad65e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 31 01:10:31.184586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.184495    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"]
Oct 31 01:10:31.187268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.187221    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:31.242034 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.242005    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-935bafe7-serving-cert\""
Oct 31 01:10:31.250892 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.249969    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"]
Oct 31 01:10:31.297483 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.297436    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96f89\" (UniqueName: \"kubernetes.io/projected/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-kube-api-access-96f89\") pod \"maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:31.297675 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.297583    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-maas-api-tls\") pod \"maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:31.297749 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.297691    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-openshift-service-ca\") pod \"maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:31.297749 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.297726    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-metrics-certs\") pod \"maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:31.398849 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.398816    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-openshift-service-ca\") pod \"maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:31.398849 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.398849    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-metrics-certs\") pod \"maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:31.399065 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.398893    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96f89\" (UniqueName: \"kubernetes.io/projected/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-kube-api-access-96f89\") pod \"maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:31.399065 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.398931    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-maas-api-tls\") pod \"maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:31.399065 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:31.399046    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-serving-cert: secret "maas-api-e2e-ait-935bafe7-serving-cert" not found
Oct 31 01:10:31.399215 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:31.399109    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-maas-api-tls podName:507f992d-fb4a-44c0-aebc-de02ce8eb9f5 nodeName:}" failed. No retries permitted until 2026-10-31 01:10:31.899091045 +0000 UTC m=+841.588974170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-maas-api-tls") pod "maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx" (UID: "507f992d-fb4a-44c0-aebc-de02ce8eb9f5") : secret "maas-api-e2e-ait-935bafe7-serving-cert" not found
Oct 31 01:10:31.399680 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.399655    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-openshift-service-ca\") pod \"maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:31.401498 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.401475    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-metrics-certs\") pod \"maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:31.454238 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.454134    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:31.457227 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.457201    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96f89\" (UniqueName: \"kubernetes.io/projected/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-kube-api-access-96f89\") pod \"maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:31.686644 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.686614    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:31.764543 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.764467    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"
Oct 31 01:10:31.820611 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.820585    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:10:31.821642 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.821622    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:10:31.904306 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.904266    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-maas-api-tls\") pod \"maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:31.907269 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.907245    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-maas-api-tls\") pod \"maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:31.907269 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:31.907272    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:32.099668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:32.099641    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:32.473624 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:32.473490    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"]
Oct 31 01:10:32.476472 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:10:32.476444    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507f992d_fb4a_44c0_aebc_de02ce8eb9f5.slice/crio-c5808fdfc0316cc710634d1efbe6e376db7efccdcbff031394cee3c4d9df619c WatchSource:0}: Error finding container c5808fdfc0316cc710634d1efbe6e376db7efccdcbff031394cee3c4d9df619c: Status 404 returned error can't find the container with id c5808fdfc0316cc710634d1efbe6e376db7efccdcbff031394cee3c4d9df619c
Oct 31 01:10:32.856138 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:32.856095    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx" event={"ID":"507f992d-fb4a-44c0-aebc-de02ce8eb9f5","Type":"ContainerStarted","Data":"c5808fdfc0316cc710634d1efbe6e376db7efccdcbff031394cee3c4d9df619c"}
Oct 31 01:10:33.670033 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:33.669953    4476 patch_prober.go:29] interesting pod/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 01:10:33.670033 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:33.670005    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 01:10:33.862726 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:33.862686    4476 generic.go:356] "Generic (PLEG): container finished" podID="e817033e-6407-4cea-88a3-ee4b801219b5" containerID="bf7da808748db6cfa121712ce4e7a9850532830c437aa47fc39bfc97e28764e5" exitCode=0
Oct 31 01:10:33.863198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:33.862766    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" event={"ID":"e817033e-6407-4cea-88a3-ee4b801219b5","Type":"ContainerDied","Data":"bf7da808748db6cfa121712ce4e7a9850532830c437aa47fc39bfc97e28764e5"}
Oct 31 01:10:33.863269 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:33.863216    4476 scope.go:122] "RemoveContainer" containerID="bf7da808748db6cfa121712ce4e7a9850532830c437aa47fc39bfc97e28764e5"
Oct 31 01:10:33.864294 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:33.864270    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx" event={"ID":"507f992d-fb4a-44c0-aebc-de02ce8eb9f5","Type":"ContainerStarted","Data":"8263772b487f122dd70d4a2eade3434027a9ff7fff75d264147ba6cce1d548da"}
Oct 31 01:10:33.864417 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:33.864404    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:33.864477 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:33.864449    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:10:33.864530 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:33.864517    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:10:33.864564 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:33.864553    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:10:34.149188 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:34.149120    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx" podStartSLOduration=2.261307217 podStartE2EDuration="3.149099915s" totalImagesPullingTime="887.792698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:34.145364935 +0000 UTC m=+843.835248064" watchObservedRunningTime="2026-10-31 01:10:34.149099915 +0000 UTC m=+843.838983043"
Oct 31 01:10:34.844401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:34.844377    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:10:34.896913 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:34.896878    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" event={"ID":"e817033e-6407-4cea-88a3-ee4b801219b5","Type":"ContainerStarted","Data":"0ef38a4aa5e6ced5f61edb3a2f95f73d46434d6f8165d31834439793280a40f4"}
Oct 31 01:10:35.189680 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:35.189589    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:35.189856 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:35.189828    4476 patch_prober.go:29] interesting pod/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 01:10:35.189934 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:35.189894    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 01:10:35.533354 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:35.533322    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:36.189768 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.189729    4476 patch_prober.go:29] interesting pod/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 01:10:36.190298 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.189797    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 01:10:36.491872 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.491789    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"]
Oct 31 01:10:36.495442 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.495426    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:36.547418 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.547385    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-60c85e15-serving-cert\""
Oct 31 01:10:36.548800 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.548781    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-metrics-certs\") pod \"maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:36.548865 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.548823    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-maas-api-tls\") pod \"maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:36.548865 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.548853    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acb2731c-cb28-48f6-8848-34838bb47a95-openshift-service-ca\") pod \"maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:36.549068 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.549047    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsc7b\" (UniqueName: \"kubernetes.io/projected/acb2731c-cb28-48f6-8848-34838bb47a95-kube-api-access-bsc7b\") pod \"maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:36.598899 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.598874    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"]
Oct 31 01:10:36.650707 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.650668    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsc7b\" (UniqueName: \"kubernetes.io/projected/acb2731c-cb28-48f6-8848-34838bb47a95-kube-api-access-bsc7b\") pod \"maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:36.650882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.650748    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-metrics-certs\") pod \"maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:36.650882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.650804    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-maas-api-tls\") pod \"maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:36.650882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.650848    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acb2731c-cb28-48f6-8848-34838bb47a95-openshift-service-ca\") pod \"maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:36.651151 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:36.651079    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-serving-cert: secret "maas-api-e2e-migrate-60c85e15-serving-cert" not found
Oct 31 01:10:36.651431 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:36.651156    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-maas-api-tls podName:acb2731c-cb28-48f6-8848-34838bb47a95 nodeName:}" failed. No retries permitted until 2026-10-31 01:10:37.151134687 +0000 UTC m=+846.841017796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-maas-api-tls") pod "maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6" (UID: "acb2731c-cb28-48f6-8848-34838bb47a95") : secret "maas-api-e2e-migrate-60c85e15-serving-cert" not found
Oct 31 01:10:36.651637 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.651614    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acb2731c-cb28-48f6-8848-34838bb47a95-openshift-service-ca\") pod \"maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:36.653757 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.653731    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-metrics-certs\") pod \"maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:36.709024 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.708990    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsc7b\" (UniqueName: \"kubernetes.io/projected/acb2731c-cb28-48f6-8848-34838bb47a95-kube-api-access-bsc7b\") pod \"maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:36.746918 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.746846    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:36.947251 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:36.947216    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:37.145629 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:37.145597    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:37.156238 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:37.156215    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-maas-api-tls\") pod \"maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:37.158973 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:37.158944    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-maas-api-tls\") pod \"maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:37.189922 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:37.189887    4476 patch_prober.go:29] interesting pod/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 01:10:37.190293 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:37.189955    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 01:10:37.433331 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:37.433248    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:37.828534 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:37.828462    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"]
Oct 31 01:10:37.831314 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:10:37.831285    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb2731c_cb28_48f6_8848_34838bb47a95.slice/crio-eed64e7e1d037f6a2d96a926a7852a5e655060d251b92843256b2fa169c07b41 WatchSource:0}: Error finding container eed64e7e1d037f6a2d96a926a7852a5e655060d251b92843256b2fa169c07b41: Status 404 returned error can't find the container with id eed64e7e1d037f6a2d96a926a7852a5e655060d251b92843256b2fa169c07b41
Oct 31 01:10:37.914846 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:37.914812    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6" event={"ID":"acb2731c-cb28-48f6-8848-34838bb47a95","Type":"ContainerStarted","Data":"eed64e7e1d037f6a2d96a926a7852a5e655060d251b92843256b2fa169c07b41"}
Oct 31 01:10:38.189931 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:38.189831    4476 patch_prober.go:29] interesting pod/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 01:10:38.189931 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:38.189893    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 01:10:38.922373 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:38.922290    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6" event={"ID":"acb2731c-cb28-48f6-8848-34838bb47a95","Type":"ContainerStarted","Data":"6e8a9a73a3b830af3d696fc92a9b6c34f4b0174a578a07b9116be5a62cb2977f"}
Oct 31 01:10:38.922517 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:38.922409    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:39.049496 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:39.049438    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6" podStartSLOduration=2.227406765 podStartE2EDuration="3.049424273s" totalImagesPullingTime="822.017508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:39.047371834 +0000 UTC m=+848.737254963" watchObservedRunningTime="2026-10-31 01:10:39.049424273 +0000 UTC m=+848.739307401"
Oct 31 01:10:39.189598 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:39.189500    4476 patch_prober.go:29] interesting pod/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 01:10:39.189598 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:39.189561    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 01:10:39.902483 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:39.902454    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:10:40.193998 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:40.193918    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:40.194309 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:40.194291    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:40.194883 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:40.194865    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:10:41.380650 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:41.380563    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:43.288914 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:43.288884    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:43.478874 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:43.478838    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:44.932492 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:44.932460    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:10:46.307232 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:46.307202    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:50.343476 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.343434    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"]
Oct 31 01:10:50.344136 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.343702    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v" podUID="57bb6d2b-e5fe-4a87-92ee-a5f9fff36602" containerName="maas-api" containerID="cri-o://b0c89fc553e56cabd79c64aefb2e5ca4d516021b1423ed32afee2a7cd0123406" gracePeriod=30
Oct 31 01:10:50.504594 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.504556    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:50.627855 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.627831    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:50.821455 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.821420    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-kube-api-access-9cc54\" (UniqueName: \"kubernetes.io/projected/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-kube-api-access-9cc54\") pod \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") "
Oct 31 01:10:50.821646 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.821468    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-maas-api-tls\") pod \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") "
Oct 31 01:10:50.821646 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.821489    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-openshift-service-ca\") pod \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") "
Oct 31 01:10:50.821646 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.821603    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-metrics-certs\") pod \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\" (UID: \"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602\") "
Oct 31 01:10:50.821932 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.821906    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-openshift-service-ca" pod "57bb6d2b-e5fe-4a87-92ee-a5f9fff36602" (UID: "57bb6d2b-e5fe-4a87-92ee-a5f9fff36602"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:10:50.823752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.823726    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-maas-api-tls" pod "57bb6d2b-e5fe-4a87-92ee-a5f9fff36602" (UID: "57bb6d2b-e5fe-4a87-92ee-a5f9fff36602"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:50.823752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.823731    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-kube-api-access-9cc54" pod "57bb6d2b-e5fe-4a87-92ee-a5f9fff36602" (UID: "57bb6d2b-e5fe-4a87-92ee-a5f9fff36602"). InnerVolumeSpecName "kube-api-access-9cc54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:50.823886 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.823754    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-metrics-certs" pod "57bb6d2b-e5fe-4a87-92ee-a5f9fff36602" (UID: "57bb6d2b-e5fe-4a87-92ee-a5f9fff36602"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:50.922422 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.922334    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:50.922422 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.922369    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cc54\" (UniqueName: \"kubernetes.io/projected/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-kube-api-access-9cc54\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:50.922422 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.922381    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:50.922422 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.922391    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:50.982334 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.982292    4476 generic.go:356] "Generic (PLEG): container finished" podID="57bb6d2b-e5fe-4a87-92ee-a5f9fff36602" containerID="b0c89fc553e56cabd79c64aefb2e5ca4d516021b1423ed32afee2a7cd0123406" exitCode=0
Oct 31 01:10:50.982535 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.982364    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"
Oct 31 01:10:50.982535 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.982371    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v" event={"ID":"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602","Type":"ContainerDied","Data":"b0c89fc553e56cabd79c64aefb2e5ca4d516021b1423ed32afee2a7cd0123406"}
Oct 31 01:10:50.982535 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.982461    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v" event={"ID":"57bb6d2b-e5fe-4a87-92ee-a5f9fff36602","Type":"ContainerDied","Data":"2b4a68689552159d80605429b0f7f2127050094ae46b9967085e88c4ea1b7b41"}
Oct 31 01:10:50.982535 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.982484    4476 scope.go:122] "RemoveContainer" containerID="b0c89fc553e56cabd79c64aefb2e5ca4d516021b1423ed32afee2a7cd0123406"
Oct 31 01:10:50.992755 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.992736    4476 scope.go:122] "RemoveContainer" containerID="b0c89fc553e56cabd79c64aefb2e5ca4d516021b1423ed32afee2a7cd0123406"
Oct 31 01:10:50.993070 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:50.993052    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c89fc553e56cabd79c64aefb2e5ca4d516021b1423ed32afee2a7cd0123406\": container with ID starting with b0c89fc553e56cabd79c64aefb2e5ca4d516021b1423ed32afee2a7cd0123406 not found: ID does not exist" containerID="b0c89fc553e56cabd79c64aefb2e5ca4d516021b1423ed32afee2a7cd0123406"
Oct 31 01:10:50.993114 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:50.993080    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c89fc553e56cabd79c64aefb2e5ca4d516021b1423ed32afee2a7cd0123406"} err="failed to get container status \"b0c89fc553e56cabd79c64aefb2e5ca4d516021b1423ed32afee2a7cd0123406\": rpc error: code = NotFound desc = could not find container \"b0c89fc553e56cabd79c64aefb2e5ca4d516021b1423ed32afee2a7cd0123406\": container with ID starting with b0c89fc553e56cabd79c64aefb2e5ca4d516021b1423ed32afee2a7cd0123406 not found: ID does not exist"
Oct 31 01:10:51.145269 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:51.145228    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"]
Oct 31 01:10:51.194902 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:51.194821    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20f0d1b6-7d764ccd66-64l4v"]
Oct 31 01:10:52.009349 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:52.009302    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:52.296547 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:52.296514    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:52.453185 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:52.453153    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57bb6d2b-e5fe-4a87-92ee-a5f9fff36602" path="/var/lib/kubelet/pods/57bb6d2b-e5fe-4a87-92ee-a5f9fff36602/volumes"
Oct 31 01:10:52.473911 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:52.473883    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:53.148724 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:53.148687    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"]
Oct 31 01:10:53.149225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:53.148937    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24" podUID="77f09923-6594-41c7-aa26-de1b020ad65e" containerName="istio-proxy" containerID="cri-o://7c1797d232d6fb5b8723574d14a9efd5df84a05ddd2e188e08b8a2eab6ed6696" gracePeriod=30
Oct 31 01:10:53.193198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:53.193158    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:54.014249 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:54.014215    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:54.667304 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:54.667267    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:54.858975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:54.858948    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:55.040874 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:55.040833    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:57.429087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:57.429049    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:58.396246 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.396221    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:10:58.496129 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496040    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-workload-socket\") pod \"77f09923-6594-41c7-aa26-de1b020ad65e\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") "
Oct 31 01:10:58.496129 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496082    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f09923-6594-41c7-aa26-de1b020ad65e-istio-token\" (UniqueName: \"kubernetes.io/projected/77f09923-6594-41c7-aa26-de1b020ad65e-istio-token\") pod \"77f09923-6594-41c7-aa26-de1b020ad65e\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") "
Oct 31 01:10:58.496129 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496117    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-credential-socket\") pod \"77f09923-6594-41c7-aa26-de1b020ad65e\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") "
Oct 31 01:10:58.496748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496156    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-istio-data\") pod \"77f09923-6594-41c7-aa26-de1b020ad65e\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") "
Oct 31 01:10:58.496748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496217    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-workload-certs\") pod \"77f09923-6594-41c7-aa26-de1b020ad65e\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") "
Oct 31 01:10:58.496748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496269    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f09923-6594-41c7-aa26-de1b020ad65e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f09923-6594-41c7-aa26-de1b020ad65e-istiod-ca-cert\") pod \"77f09923-6594-41c7-aa26-de1b020ad65e\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") "
Oct 31 01:10:58.496748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496293    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f09923-6594-41c7-aa26-de1b020ad65e-kube-api-access-tnjcr\" (UniqueName: \"kubernetes.io/projected/77f09923-6594-41c7-aa26-de1b020ad65e-kube-api-access-tnjcr\") pod \"77f09923-6594-41c7-aa26-de1b020ad65e\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") "
Oct 31 01:10:58.496748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496351    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77f09923-6594-41c7-aa26-de1b020ad65e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f09923-6594-41c7-aa26-de1b020ad65e-istio-podinfo\") pod \"77f09923-6594-41c7-aa26-de1b020ad65e\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") "
Oct 31 01:10:58.496748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496391    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-istio-envoy\") pod \"77f09923-6594-41c7-aa26-de1b020ad65e\" (UID: \"77f09923-6594-41c7-aa26-de1b020ad65e\") "
Oct 31 01:10:58.496748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496412    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-workload-socket" pod "77f09923-6594-41c7-aa26-de1b020ad65e" (UID: "77f09923-6594-41c7-aa26-de1b020ad65e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:58.496748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496550    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-credential-socket" pod "77f09923-6594-41c7-aa26-de1b020ad65e" (UID: "77f09923-6594-41c7-aa26-de1b020ad65e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:58.496748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496699    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-workload-certs" pod "77f09923-6594-41c7-aa26-de1b020ad65e" (UID: "77f09923-6594-41c7-aa26-de1b020ad65e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:58.496748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496712    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-istio-data" pod "77f09923-6594-41c7-aa26-de1b020ad65e" (UID: "77f09923-6594-41c7-aa26-de1b020ad65e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:58.497204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496801    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:58.497204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496822    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:58.497204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496838    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:58.497204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496853    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:58.497204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.496918    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f09923-6594-41c7-aa26-de1b020ad65e-istiod-ca-cert" pod "77f09923-6594-41c7-aa26-de1b020ad65e" (UID: "77f09923-6594-41c7-aa26-de1b020ad65e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:10:58.498784 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.498757    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77f09923-6594-41c7-aa26-de1b020ad65e-istio-podinfo" pod "77f09923-6594-41c7-aa26-de1b020ad65e" (UID: "77f09923-6594-41c7-aa26-de1b020ad65e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:10:58.498908 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.498799    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f09923-6594-41c7-aa26-de1b020ad65e-kube-api-access-tnjcr" pod "77f09923-6594-41c7-aa26-de1b020ad65e" (UID: "77f09923-6594-41c7-aa26-de1b020ad65e"). InnerVolumeSpecName "kube-api-access-tnjcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:58.499078 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.499062    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-istio-envoy" pod "77f09923-6594-41c7-aa26-de1b020ad65e" (UID: "77f09923-6594-41c7-aa26-de1b020ad65e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:58.499137 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.499077    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f09923-6594-41c7-aa26-de1b020ad65e-istio-token" pod "77f09923-6594-41c7-aa26-de1b020ad65e" (UID: "77f09923-6594-41c7-aa26-de1b020ad65e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:58.598003 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.597969    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f09923-6594-41c7-aa26-de1b020ad65e-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:58.598003 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.598003    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f09923-6594-41c7-aa26-de1b020ad65e-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:58.598267 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.598016    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f09923-6594-41c7-aa26-de1b020ad65e-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:58.598267 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.598029    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f09923-6594-41c7-aa26-de1b020ad65e-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:58.598267 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:58.598041    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnjcr\" (UniqueName: \"kubernetes.io/projected/77f09923-6594-41c7-aa26-de1b020ad65e-kube-api-access-tnjcr\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:59.024110 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:59.024073    4476 generic.go:356] "Generic (PLEG): container finished" podID="77f09923-6594-41c7-aa26-de1b020ad65e" containerID="7c1797d232d6fb5b8723574d14a9efd5df84a05ddd2e188e08b8a2eab6ed6696" exitCode=0
Oct 31 01:10:59.024325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:59.024123    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"
Oct 31 01:10:59.024325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:59.024156    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24" event={"ID":"77f09923-6594-41c7-aa26-de1b020ad65e","Type":"ContainerDied","Data":"7c1797d232d6fb5b8723574d14a9efd5df84a05ddd2e188e08b8a2eab6ed6696"}
Oct 31 01:10:59.024325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:59.024234    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24" event={"ID":"77f09923-6594-41c7-aa26-de1b020ad65e","Type":"ContainerDied","Data":"fcc43fc6904c696e9c8b1cea1a178058ed36643ca334a94192a97415cbbf2a06"}
Oct 31 01:10:59.024325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:59.024259    4476 scope.go:122] "RemoveContainer" containerID="7c1797d232d6fb5b8723574d14a9efd5df84a05ddd2e188e08b8a2eab6ed6696"
Oct 31 01:10:59.033647 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:59.033620    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:59.035067 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:59.035044    4476 scope.go:122] "RemoveContainer" containerID="8f67e85ea963b57959c0b000a3da71a9cacdf3dbc2bd54cbc53fd545b679f9e4"
Oct 31 01:10:59.043385 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:59.043369    4476 scope.go:122] "RemoveContainer" containerID="7c1797d232d6fb5b8723574d14a9efd5df84a05ddd2e188e08b8a2eab6ed6696"
Oct 31 01:10:59.043660 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:59.043643    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c1797d232d6fb5b8723574d14a9efd5df84a05ddd2e188e08b8a2eab6ed6696\": container with ID starting with 7c1797d232d6fb5b8723574d14a9efd5df84a05ddd2e188e08b8a2eab6ed6696 not found: ID does not exist" containerID="7c1797d232d6fb5b8723574d14a9efd5df84a05ddd2e188e08b8a2eab6ed6696"
Oct 31 01:10:59.043730 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:59.043670    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c1797d232d6fb5b8723574d14a9efd5df84a05ddd2e188e08b8a2eab6ed6696"} err="failed to get container status \"7c1797d232d6fb5b8723574d14a9efd5df84a05ddd2e188e08b8a2eab6ed6696\": rpc error: code = NotFound desc = could not find container \"7c1797d232d6fb5b8723574d14a9efd5df84a05ddd2e188e08b8a2eab6ed6696\": container with ID starting with 7c1797d232d6fb5b8723574d14a9efd5df84a05ddd2e188e08b8a2eab6ed6696 not found: ID does not exist"
Oct 31 01:10:59.043730 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:59.043685    4476 scope.go:122] "RemoveContainer" containerID="8f67e85ea963b57959c0b000a3da71a9cacdf3dbc2bd54cbc53fd545b679f9e4"
Oct 31 01:10:59.043912 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:59.043895    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f67e85ea963b57959c0b000a3da71a9cacdf3dbc2bd54cbc53fd545b679f9e4\": container with ID starting with 8f67e85ea963b57959c0b000a3da71a9cacdf3dbc2bd54cbc53fd545b679f9e4 not found: ID does not exist" containerID="8f67e85ea963b57959c0b000a3da71a9cacdf3dbc2bd54cbc53fd545b679f9e4"
Oct 31 01:10:59.043952 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:59.043917    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f67e85ea963b57959c0b000a3da71a9cacdf3dbc2bd54cbc53fd545b679f9e4"} err="failed to get container status \"8f67e85ea963b57959c0b000a3da71a9cacdf3dbc2bd54cbc53fd545b679f9e4\": rpc error: code = NotFound desc = could not find container \"8f67e85ea963b57959c0b000a3da71a9cacdf3dbc2bd54cbc53fd545b679f9e4\": container with ID starting with 8f67e85ea963b57959c0b000a3da71a9cacdf3dbc2bd54cbc53fd545b679f9e4 not found: ID does not exist"
Oct 31 01:10:59.182245 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:10:59.182209    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:10:59.202525 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:59.202495    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"]
Oct 31 01:10:59.251821 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:59.251788    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-20f0d1b6-openshift-default-567d5c7dc-g7s24"]
Oct 31 01:10:59.255900 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:10:59.255879    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:00.453136 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:00.453107    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f09923-6594-41c7-aa26-de1b020ad65e" path="/var/lib/kubelet/pods/77f09923-6594-41c7-aa26-de1b020ad65e/volumes"
Oct 31 01:11:02.492126 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:02.492092    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:02.979960 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:02.979930    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:03.165191 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:03.165145    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:07.699290 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:07.699252    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:09.853773 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:09.853720    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"]
Oct 31 01:11:09.958901 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:09.958869    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"]
Oct 31 01:11:10.256896 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:10.256863    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:10.442549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:10.442469    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:12.029390 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.029357    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"]
Oct 31 01:11:12.029905 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.029609    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd" podUID="3cb82634-defb-47d4-8f9b-eebcc042d6f0" containerName="maas-api" containerID="cri-o://cc7022de4b2e8e53e9b72c12d25a0182002c679bb3ca76e36a12c2d9eef2af54" gracePeriod=30
Oct 31 01:11:12.178908 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.178876    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:12.291022 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.290956    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:11:12.429581 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.429548    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb82634-defb-47d4-8f9b-eebcc042d6f0-kube-api-access-96ch8\" (UniqueName: \"kubernetes.io/projected/3cb82634-defb-47d4-8f9b-eebcc042d6f0-kube-api-access-96ch8\") pod \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") "
Oct 31 01:11:12.429765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.429639    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb82634-defb-47d4-8f9b-eebcc042d6f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb82634-defb-47d4-8f9b-eebcc042d6f0-openshift-service-ca\") pod \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") "
Oct 31 01:11:12.429765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.429707    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-maas-api-tls\") pod \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") "
Oct 31 01:11:12.429897 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.429787    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-metrics-certs\") pod \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\" (UID: \"3cb82634-defb-47d4-8f9b-eebcc042d6f0\") "
Oct 31 01:11:12.430059 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.430034    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb82634-defb-47d4-8f9b-eebcc042d6f0-openshift-service-ca" pod "3cb82634-defb-47d4-8f9b-eebcc042d6f0" (UID: "3cb82634-defb-47d4-8f9b-eebcc042d6f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:12.430145 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.430117    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb82634-defb-47d4-8f9b-eebcc042d6f0-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:12.431932 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.431896    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-maas-api-tls" pod "3cb82634-defb-47d4-8f9b-eebcc042d6f0" (UID: "3cb82634-defb-47d4-8f9b-eebcc042d6f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:12.431932 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.431916    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb82634-defb-47d4-8f9b-eebcc042d6f0-kube-api-access-96ch8" pod "3cb82634-defb-47d4-8f9b-eebcc042d6f0" (UID: "3cb82634-defb-47d4-8f9b-eebcc042d6f0"). InnerVolumeSpecName "kube-api-access-96ch8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:12.432073 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.431990    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-metrics-certs" pod "3cb82634-defb-47d4-8f9b-eebcc042d6f0" (UID: "3cb82634-defb-47d4-8f9b-eebcc042d6f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:12.531243 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.531212    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:12.531243 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.531237    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cb82634-defb-47d4-8f9b-eebcc042d6f0-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:12.531243 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.531250    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96ch8\" (UniqueName: \"kubernetes.io/projected/3cb82634-defb-47d4-8f9b-eebcc042d6f0-kube-api-access-96ch8\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:12.680456 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.680378    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:12.864297 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:12.864268    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:13.101451 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.101421    4476 generic.go:356] "Generic (PLEG): container finished" podID="3cb82634-defb-47d4-8f9b-eebcc042d6f0" containerID="cc7022de4b2e8e53e9b72c12d25a0182002c679bb3ca76e36a12c2d9eef2af54" exitCode=0
Oct 31 01:11:13.101924 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.101468    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"
Oct 31 01:11:13.101924 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.101513    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd" event={"ID":"3cb82634-defb-47d4-8f9b-eebcc042d6f0","Type":"ContainerDied","Data":"cc7022de4b2e8e53e9b72c12d25a0182002c679bb3ca76e36a12c2d9eef2af54"}
Oct 31 01:11:13.101924 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.101563    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd" event={"ID":"3cb82634-defb-47d4-8f9b-eebcc042d6f0","Type":"ContainerDied","Data":"c6bce67fac8dfca830542d6b9dbfb9a80c549f32dc7e370115adabcff6d1a8df"}
Oct 31 01:11:13.101924 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.101585    4476 scope.go:122] "RemoveContainer" containerID="cc7022de4b2e8e53e9b72c12d25a0182002c679bb3ca76e36a12c2d9eef2af54"
Oct 31 01:11:13.110530 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.110515    4476 scope.go:122] "RemoveContainer" containerID="cc7022de4b2e8e53e9b72c12d25a0182002c679bb3ca76e36a12c2d9eef2af54"
Oct 31 01:11:13.110789 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:13.110772    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7022de4b2e8e53e9b72c12d25a0182002c679bb3ca76e36a12c2d9eef2af54\": container with ID starting with cc7022de4b2e8e53e9b72c12d25a0182002c679bb3ca76e36a12c2d9eef2af54 not found: ID does not exist" containerID="cc7022de4b2e8e53e9b72c12d25a0182002c679bb3ca76e36a12c2d9eef2af54"
Oct 31 01:11:13.110848 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.110798    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7022de4b2e8e53e9b72c12d25a0182002c679bb3ca76e36a12c2d9eef2af54"} err="failed to get container status \"cc7022de4b2e8e53e9b72c12d25a0182002c679bb3ca76e36a12c2d9eef2af54\": rpc error: code = NotFound desc = could not find container \"cc7022de4b2e8e53e9b72c12d25a0182002c679bb3ca76e36a12c2d9eef2af54\": container with ID starting with cc7022de4b2e8e53e9b72c12d25a0182002c679bb3ca76e36a12c2d9eef2af54 not found: ID does not exist"
Oct 31 01:11:13.166094 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.166062    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"]
Oct 31 01:11:13.166294 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.166275    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg" podUID="6b46f41d-5a73-49ed-a5a0-a4c4d041f86b" containerName="maas-api" containerID="cri-o://5cc2f875d1e7481c1532c1dc0ede758396b81871a588da047bd511124139f5ce" gracePeriod=30
Oct 31 01:11:13.273931 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.273890    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"]
Oct 31 01:11:13.274193 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.274158    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-79f7219a-79b4ddfc5b-b4vtd"]
Oct 31 01:11:13.316070 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.316031    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:13.424161 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.424131    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:11:13.540244 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.540209    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-metrics-certs\") pod \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") "
Oct 31 01:11:13.540456 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.540401    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-maas-api-tls\") pod \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") "
Oct 31 01:11:13.540456 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.540442    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-kube-api-access-xzkwh\" (UniqueName: \"kubernetes.io/projected/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-kube-api-access-xzkwh\") pod \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") "
Oct 31 01:11:13.540573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.540491    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-openshift-service-ca\") pod \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\" (UID: \"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b\") "
Oct 31 01:11:13.542924 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.541279    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-openshift-service-ca" pod "6b46f41d-5a73-49ed-a5a0-a4c4d041f86b" (UID: "6b46f41d-5a73-49ed-a5a0-a4c4d041f86b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:13.543162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.543138    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-metrics-certs" pod "6b46f41d-5a73-49ed-a5a0-a4c4d041f86b" (UID: "6b46f41d-5a73-49ed-a5a0-a4c4d041f86b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:13.543296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.543272    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-kube-api-access-xzkwh" pod "6b46f41d-5a73-49ed-a5a0-a4c4d041f86b" (UID: "6b46f41d-5a73-49ed-a5a0-a4c4d041f86b"). InnerVolumeSpecName "kube-api-access-xzkwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:13.543589 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.543565    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-maas-api-tls" pod "6b46f41d-5a73-49ed-a5a0-a4c4d041f86b" (UID: "6b46f41d-5a73-49ed-a5a0-a4c4d041f86b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:13.641862 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.641828    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:13.641862 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.641856    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzkwh\" (UniqueName: \"kubernetes.io/projected/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-kube-api-access-xzkwh\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:13.641862 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.641867    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:13.642097 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:13.641875    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:14.110225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:14.110163    4476 generic.go:356] "Generic (PLEG): container finished" podID="6b46f41d-5a73-49ed-a5a0-a4c4d041f86b" containerID="5cc2f875d1e7481c1532c1dc0ede758396b81871a588da047bd511124139f5ce" exitCode=0
Oct 31 01:11:14.110225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:14.110207    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg" event={"ID":"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b","Type":"ContainerDied","Data":"5cc2f875d1e7481c1532c1dc0ede758396b81871a588da047bd511124139f5ce"}
Oct 31 01:11:14.110808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:14.110246    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"
Oct 31 01:11:14.110808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:14.110308    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg" event={"ID":"6b46f41d-5a73-49ed-a5a0-a4c4d041f86b","Type":"ContainerDied","Data":"f5e253ab3c385f8564660e0e9af32f71a070bb434e3c003d7dd4cffd16019648"}
Oct 31 01:11:14.110808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:14.110339    4476 scope.go:122] "RemoveContainer" containerID="5cc2f875d1e7481c1532c1dc0ede758396b81871a588da047bd511124139f5ce"
Oct 31 01:11:14.120896 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:14.120878    4476 scope.go:122] "RemoveContainer" containerID="5cc2f875d1e7481c1532c1dc0ede758396b81871a588da047bd511124139f5ce"
Oct 31 01:11:14.121321 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:14.121256    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc2f875d1e7481c1532c1dc0ede758396b81871a588da047bd511124139f5ce\": container with ID starting with 5cc2f875d1e7481c1532c1dc0ede758396b81871a588da047bd511124139f5ce not found: ID does not exist" containerID="5cc2f875d1e7481c1532c1dc0ede758396b81871a588da047bd511124139f5ce"
Oct 31 01:11:14.121321 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:14.121290    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc2f875d1e7481c1532c1dc0ede758396b81871a588da047bd511124139f5ce"} err="failed to get container status \"5cc2f875d1e7481c1532c1dc0ede758396b81871a588da047bd511124139f5ce\": rpc error: code = NotFound desc = could not find container \"5cc2f875d1e7481c1532c1dc0ede758396b81871a588da047bd511124139f5ce\": container with ID starting with 5cc2f875d1e7481c1532c1dc0ede758396b81871a588da047bd511124139f5ce not found: ID does not exist"
Oct 31 01:11:14.273012 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:14.272978    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"]
Oct 31 01:11:14.322604 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:14.322571    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3d4f9b87-5fd64ff86b-q5whg"]
Oct 31 01:11:14.452772 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:14.452681    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb82634-defb-47d4-8f9b-eebcc042d6f0" path="/var/lib/kubelet/pods/3cb82634-defb-47d4-8f9b-eebcc042d6f0/volumes"
Oct 31 01:11:14.453139 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:14.453120    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b46f41d-5a73-49ed-a5a0-a4c4d041f86b" path="/var/lib/kubelet/pods/6b46f41d-5a73-49ed-a5a0-a4c4d041f86b/volumes"
Oct 31 01:11:14.854358 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:14.854317    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv" podUID="50972db7-acce-465f-899d-048b1662e9e6" containerName="payload-processing" containerID="cri-o://bb245543868c4c1dc86ee80afb43436256be88529b31ae0e02ae73681e4c8429" gracePeriod=25
Oct 31 01:11:14.959465 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:14.959421    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf" podUID="683400b4-343f-4699-bc65-30e2230105c1" containerName="payload-pre-processing" containerID="cri-o://7f65b4cfdb5be2815af4e2cd9594107302283e5000bf38b3497d3466603dc411" gracePeriod=25
Oct 31 01:11:15.079162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.079131    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:15.114121 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.114039    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"
Oct 31 01:11:15.116878 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.116849    4476 generic.go:356] "Generic (PLEG): container finished" podID="50972db7-acce-465f-899d-048b1662e9e6" containerID="bb245543868c4c1dc86ee80afb43436256be88529b31ae0e02ae73681e4c8429" exitCode=0
Oct 31 01:11:15.117024 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.116895    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"
Oct 31 01:11:15.117024 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.116923    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv" event={"ID":"50972db7-acce-465f-899d-048b1662e9e6","Type":"ContainerDied","Data":"bb245543868c4c1dc86ee80afb43436256be88529b31ae0e02ae73681e4c8429"}
Oct 31 01:11:15.117024 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.116990    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv" event={"ID":"50972db7-acce-465f-899d-048b1662e9e6","Type":"ContainerDied","Data":"d0c308bd7b01e8814a400ac13a8df49e72d5dbd67c6df5c04d8a9d1207434394"}
Oct 31 01:11:15.117024 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.117019    4476 scope.go:122] "RemoveContainer" containerID="bb245543868c4c1dc86ee80afb43436256be88529b31ae0e02ae73681e4c8429"
Oct 31 01:11:15.120249 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.120223    4476 generic.go:356] "Generic (PLEG): container finished" podID="683400b4-343f-4699-bc65-30e2230105c1" containerID="7f65b4cfdb5be2815af4e2cd9594107302283e5000bf38b3497d3466603dc411" exitCode=0
Oct 31 01:11:15.120365 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.120277    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf" event={"ID":"683400b4-343f-4699-bc65-30e2230105c1","Type":"ContainerDied","Data":"7f65b4cfdb5be2815af4e2cd9594107302283e5000bf38b3497d3466603dc411"}
Oct 31 01:11:15.126349 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.126332    4476 scope.go:122] "RemoveContainer" containerID="bb245543868c4c1dc86ee80afb43436256be88529b31ae0e02ae73681e4c8429"
Oct 31 01:11:15.126633 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:15.126614    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb245543868c4c1dc86ee80afb43436256be88529b31ae0e02ae73681e4c8429\": container with ID starting with bb245543868c4c1dc86ee80afb43436256be88529b31ae0e02ae73681e4c8429 not found: ID does not exist" containerID="bb245543868c4c1dc86ee80afb43436256be88529b31ae0e02ae73681e4c8429"
Oct 31 01:11:15.126692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.126639    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb245543868c4c1dc86ee80afb43436256be88529b31ae0e02ae73681e4c8429"} err="failed to get container status \"bb245543868c4c1dc86ee80afb43436256be88529b31ae0e02ae73681e4c8429\": rpc error: code = NotFound desc = could not find container \"bb245543868c4c1dc86ee80afb43436256be88529b31ae0e02ae73681e4c8429\": container with ID starting with bb245543868c4c1dc86ee80afb43436256be88529b31ae0e02ae73681e4c8429 not found: ID does not exist"
Oct 31 01:11:15.224827 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.224798    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"
Oct 31 01:11:15.258255 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.258227    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50972db7-acce-465f-899d-048b1662e9e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50972db7-acce-465f-899d-048b1662e9e6-plugins-config-volume\") pod \"50972db7-acce-465f-899d-048b1662e9e6\" (UID: \"50972db7-acce-465f-899d-048b1662e9e6\") "
Oct 31 01:11:15.258405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.258271    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50972db7-acce-465f-899d-048b1662e9e6-kube-api-access-qsnf9\" (UniqueName: \"kubernetes.io/projected/50972db7-acce-465f-899d-048b1662e9e6-kube-api-access-qsnf9\") pod \"50972db7-acce-465f-899d-048b1662e9e6\" (UID: \"50972db7-acce-465f-899d-048b1662e9e6\") "
Oct 31 01:11:15.258722 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.258696    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50972db7-acce-465f-899d-048b1662e9e6-plugins-config-volume" pod "50972db7-acce-465f-899d-048b1662e9e6" (UID: "50972db7-acce-465f-899d-048b1662e9e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:15.260537 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.260518    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50972db7-acce-465f-899d-048b1662e9e6-kube-api-access-qsnf9" pod "50972db7-acce-465f-899d-048b1662e9e6" (UID: "50972db7-acce-465f-899d-048b1662e9e6"). InnerVolumeSpecName "kube-api-access-qsnf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:15.359719 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.359691    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/683400b4-343f-4699-bc65-30e2230105c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683400b4-343f-4699-bc65-30e2230105c1-plugins-config-volume\") pod \"683400b4-343f-4699-bc65-30e2230105c1\" (UID: \"683400b4-343f-4699-bc65-30e2230105c1\") "
Oct 31 01:11:15.359904 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.359752    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683400b4-343f-4699-bc65-30e2230105c1-kube-api-access-5j8jw\" (UniqueName: \"kubernetes.io/projected/683400b4-343f-4699-bc65-30e2230105c1-kube-api-access-5j8jw\") pod \"683400b4-343f-4699-bc65-30e2230105c1\" (UID: \"683400b4-343f-4699-bc65-30e2230105c1\") "
Oct 31 01:11:15.360005 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.359989    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsnf9\" (UniqueName: \"kubernetes.io/projected/50972db7-acce-465f-899d-048b1662e9e6-kube-api-access-qsnf9\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:15.360047 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.360009    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50972db7-acce-465f-899d-048b1662e9e6-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:15.360089 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.360067    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683400b4-343f-4699-bc65-30e2230105c1-plugins-config-volume" pod "683400b4-343f-4699-bc65-30e2230105c1" (UID: "683400b4-343f-4699-bc65-30e2230105c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:15.361936 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.361909    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683400b4-343f-4699-bc65-30e2230105c1-kube-api-access-5j8jw" pod "683400b4-343f-4699-bc65-30e2230105c1" (UID: "683400b4-343f-4699-bc65-30e2230105c1"). InnerVolumeSpecName "kube-api-access-5j8jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:15.437933 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.437899    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:15.461317 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.461281    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683400b4-343f-4699-bc65-30e2230105c1-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:15.461317 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.461310    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j8jw\" (UniqueName: \"kubernetes.io/projected/683400b4-343f-4699-bc65-30e2230105c1-kube-api-access-5j8jw\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:15.535465 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.535432    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"]
Oct 31 01:11:15.541453 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:15.541431    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-79f7219a-5ffd85f9-gp4cv"]
Oct 31 01:11:16.127886 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:16.127802    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf" event={"ID":"683400b4-343f-4699-bc65-30e2230105c1","Type":"ContainerDied","Data":"c461c603f1f48272e4e9a0522998a826e3046a01a1f286578dfa264015799ba2"}
Oct 31 01:11:16.127886 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:16.127860    4476 scope.go:122] "RemoveContainer" containerID="7f65b4cfdb5be2815af4e2cd9594107302283e5000bf38b3497d3466603dc411"
Oct 31 01:11:16.127886 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:16.127817    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"
Oct 31 01:11:16.289686 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:16.289645    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"]
Oct 31 01:11:16.311542 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:16.307106    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-79f7219a-58dcf45b9f-kbdvf"]
Oct 31 01:11:16.453280 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:16.453170    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50972db7-acce-465f-899d-048b1662e9e6" path="/var/lib/kubelet/pods/50972db7-acce-465f-899d-048b1662e9e6/volumes"
Oct 31 01:11:16.453587 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:16.453574    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="683400b4-343f-4699-bc65-30e2230105c1" path="/var/lib/kubelet/pods/683400b4-343f-4699-bc65-30e2230105c1/volumes"
Oct 31 01:11:19.486288 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486252    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"]
Oct 31 01:11:19.486673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486623    4476 cpu_manager.go:414] "containerMap: removing container" podUID="77f09923-6594-41c7-aa26-de1b020ad65e" containerName="istio-proxy"
Oct 31 01:11:19.486673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486634    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f09923-6594-41c7-aa26-de1b020ad65e" containerName="istio-proxy"
Oct 31 01:11:19.486673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486649    4476 cpu_manager.go:414] "containerMap: removing container" podUID="50972db7-acce-465f-899d-048b1662e9e6" containerName="payload-processing"
Oct 31 01:11:19.486673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486655    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50972db7-acce-465f-899d-048b1662e9e6" containerName="payload-processing"
Oct 31 01:11:19.486673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486663    4476 cpu_manager.go:414] "containerMap: removing container" podUID="683400b4-343f-4699-bc65-30e2230105c1" containerName="payload-pre-processing"
Oct 31 01:11:19.486673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486668    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683400b4-343f-4699-bc65-30e2230105c1" containerName="payload-pre-processing"
Oct 31 01:11:19.486867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486687    4476 cpu_manager.go:414] "containerMap: removing container" podUID="77f09923-6594-41c7-aa26-de1b020ad65e" containerName="istio-proxy"
Oct 31 01:11:19.486867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486692    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f09923-6594-41c7-aa26-de1b020ad65e" containerName="istio-proxy"
Oct 31 01:11:19.486867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486701    4476 cpu_manager.go:414] "containerMap: removing container" podUID="6b46f41d-5a73-49ed-a5a0-a4c4d041f86b" containerName="maas-api"
Oct 31 01:11:19.486867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486705    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b46f41d-5a73-49ed-a5a0-a4c4d041f86b" containerName="maas-api"
Oct 31 01:11:19.486867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486711    4476 cpu_manager.go:414] "containerMap: removing container" podUID="57bb6d2b-e5fe-4a87-92ee-a5f9fff36602" containerName="maas-api"
Oct 31 01:11:19.486867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486717    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57bb6d2b-e5fe-4a87-92ee-a5f9fff36602" containerName="maas-api"
Oct 31 01:11:19.486867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486723    4476 cpu_manager.go:414] "containerMap: removing container" podUID="3cb82634-defb-47d4-8f9b-eebcc042d6f0" containerName="maas-api"
Oct 31 01:11:19.486867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486727    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb82634-defb-47d4-8f9b-eebcc042d6f0" containerName="maas-api"
Oct 31 01:11:19.486867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486780    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f09923-6594-41c7-aa26-de1b020ad65e" containerName="istio-proxy"
Oct 31 01:11:19.486867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486790    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="50972db7-acce-465f-899d-048b1662e9e6" containerName="payload-processing"
Oct 31 01:11:19.486867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486796    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="683400b4-343f-4699-bc65-30e2230105c1" containerName="payload-pre-processing"
Oct 31 01:11:19.486867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486805    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b46f41d-5a73-49ed-a5a0-a4c4d041f86b" containerName="maas-api"
Oct 31 01:11:19.486867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486810    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="57bb6d2b-e5fe-4a87-92ee-a5f9fff36602" containerName="maas-api"
Oct 31 01:11:19.486867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.486815    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb82634-defb-47d4-8f9b-eebcc042d6f0" containerName="maas-api"
Oct 31 01:11:19.491026 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.491010    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:19.597471 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.597441    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-26fbc4-serving-cert\""
Oct 31 01:11:19.599413 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.599390    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"]
Oct 31 01:11:19.603328 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.603307    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bsx6\" (UniqueName: \"kubernetes.io/projected/ef906818-75f2-40e3-a4b1-16f6f5e40a70-kube-api-access-2bsx6\") pod \"maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:19.603397 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.603361    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-maas-api-tls\") pod \"maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:19.603546 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.603529    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-metrics-certs\") pod \"maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:19.603612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.603597    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef906818-75f2-40e3-a4b1-16f6f5e40a70-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:19.704471 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.704437    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-maas-api-tls\") pod \"maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:19.704655 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.704507    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-metrics-certs\") pod \"maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:19.704655 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.704552    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef906818-75f2-40e3-a4b1-16f6f5e40a70-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:19.704655 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.704597    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bsx6\" (UniqueName: \"kubernetes.io/projected/ef906818-75f2-40e3-a4b1-16f6f5e40a70-kube-api-access-2bsx6\") pod \"maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:19.704655 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:19.704604    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-serving-cert: secret "maas-api-e2e-worker-w0-26fbc4-serving-cert" not found
Oct 31 01:11:19.704795 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:19.704696    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-maas-api-tls podName:ef906818-75f2-40e3-a4b1-16f6f5e40a70 nodeName:}" failed. No retries permitted until 2026-10-31 01:11:20.204673723 +0000 UTC m=+889.894556853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-maas-api-tls") pod "maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk" (UID: "ef906818-75f2-40e3-a4b1-16f6f5e40a70") : secret "maas-api-e2e-worker-w0-26fbc4-serving-cert" not found
Oct 31 01:11:19.705341 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.705315    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef906818-75f2-40e3-a4b1-16f6f5e40a70-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:19.707070 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.707052    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-metrics-certs\") pod \"maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:19.729137 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.729102    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:19.763970 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.763891    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bsx6\" (UniqueName: \"kubernetes.io/projected/ef906818-75f2-40e3-a4b1-16f6f5e40a70-kube-api-access-2bsx6\") pod \"maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:19.926704 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:19.926671    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:20.208286 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:20.208247    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-maas-api-tls\") pod \"maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:20.210928 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:20.210902    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-maas-api-tls\") pod \"maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:20.403543 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:20.403503    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:20.432345 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:20.432315    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:20.685835 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:20.685799    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:20.779931 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:20.779906    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"]
Oct 31 01:11:20.782604 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:11:20.782578    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef906818_75f2_40e3_a4b1_16f6f5e40a70.slice/crio-cafcaa8425375538f8bb6b4718b86fea51fbc1a3bfe77180ec84bdee9482380c WatchSource:0}: Error finding container cafcaa8425375538f8bb6b4718b86fea51fbc1a3bfe77180ec84bdee9482380c: Status 404 returned error can't find the container with id cafcaa8425375538f8bb6b4718b86fea51fbc1a3bfe77180ec84bdee9482380c
Oct 31 01:11:21.154440 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:21.154402    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk" event={"ID":"ef906818-75f2-40e3-a4b1-16f6f5e40a70","Type":"ContainerStarted","Data":"cafcaa8425375538f8bb6b4718b86fea51fbc1a3bfe77180ec84bdee9482380c"}
Oct 31 01:11:22.161579 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:22.161541    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk" event={"ID":"ef906818-75f2-40e3-a4b1-16f6f5e40a70","Type":"ContainerStarted","Data":"e26a242f08870ac6e7b747ad82767d94393e133ee0bb06bbf2498e88fb4ea809"}
Oct 31 01:11:22.162008 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:22.161603    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:22.279646 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:22.279594    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk" podStartSLOduration=2.392918535 podStartE2EDuration="3.279578915s" totalImagesPullingTime="886.66038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:22.278235395 +0000 UTC m=+891.968118524" watchObservedRunningTime="2026-10-31 01:11:22.279578915 +0000 UTC m=+891.969462042"
Oct 31 01:11:22.329718 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:22.329679    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"]
Oct 31 01:11:22.468909 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:22.468813    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:22.686721 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:22.686676    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:22.987241 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:22.987210    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:23.276586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.276491    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:23.777372 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.777337    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"]
Oct 31 01:11:23.777823 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.777809    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f09923-6594-41c7-aa26-de1b020ad65e" containerName="istio-proxy"
Oct 31 01:11:23.781053 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.781035    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:23.886109 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.886078    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-acd385-openshift-default-dockercfg-76xk7\""
Oct 31 01:11:23.888329 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.888306    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"]
Oct 31 01:11:23.927868 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.927839    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:23.944268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.944242    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:23.944400 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.944274    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:23.944400 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.944292    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:23.944400 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.944319    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43eb8302-cba6-4f09-a38c-d548c01abb46-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:23.944504 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.944427    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:23.944504 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.944471    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:23.944581 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.944565    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:23.944617 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.944598    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:23.944650 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:23.944622    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr99d\" (UniqueName: \"kubernetes.io/projected/43eb8302-cba6-4f09-a38c-d548c01abb46-kube-api-access-fr99d\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.045441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.045414    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.045441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.045452    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.045705 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.045476    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr99d\" (UniqueName: \"kubernetes.io/projected/43eb8302-cba6-4f09-a38c-d548c01abb46-kube-api-access-fr99d\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.045705 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.045521    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.045705 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.045542    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.045705 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.045558    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.045705 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.045595    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43eb8302-cba6-4f09-a38c-d548c01abb46-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.045705 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.045640    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.045955 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.045687    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.045955 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.045859    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.046206 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.046152    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.046306 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.046249    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.046306 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.046286    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.046377 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.046356    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43eb8302-cba6-4f09-a38c-d548c01abb46-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.047975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.047944    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.048077 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.048013    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.102273 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.102242    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.102462 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.102443    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr99d\" (UniqueName: \"kubernetes.io/projected/43eb8302-cba6-4f09-a38c-d548c01abb46-kube-api-access-fr99d\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.117875 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.117847    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:24.393474 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.393390    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:24.815023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.814999    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"]
Oct 31 01:11:24.816628 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:11:24.816598    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43eb8302_cba6_4f09_a38c_d548c01abb46.slice/crio-4b469c33cad4af8f68e6b7d4bac3c349553e3cd712fc9ea9055ed13769728e4b WatchSource:0}: Error finding container 4b469c33cad4af8f68e6b7d4bac3c349553e3cd712fc9ea9055ed13769728e4b: Status 404 returned error can't find the container with id 4b469c33cad4af8f68e6b7d4bac3c349553e3cd712fc9ea9055ed13769728e4b
Oct 31 01:11:24.818908 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.818877    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:24.818990 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.818938    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:24.818990 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:24.818967    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:25.016632 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:25.016599    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:25.178596 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:25.178502    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" event={"ID":"43eb8302-cba6-4f09-a38c-d548c01abb46","Type":"ContainerStarted","Data":"98b36cc107b9f5365f69d25639d4932f3ec13182c2273fb057ff950bedea398c"}
Oct 31 01:11:25.178596 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:25.178562    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" event={"ID":"43eb8302-cba6-4f09-a38c-d548c01abb46","Type":"ContainerStarted","Data":"4b469c33cad4af8f68e6b7d4bac3c349553e3cd712fc9ea9055ed13769728e4b"}
Oct 31 01:11:25.294026 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:25.293947    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" podStartSLOduration=2.293929477 podStartE2EDuration="2.293929477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:25.293425254 +0000 UTC m=+894.983308375" watchObservedRunningTime="2026-10-31 01:11:25.293929477 +0000 UTC m=+894.983812604"
Oct 31 01:11:25.393834 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:25.393801    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:25.399458 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:25.399434    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:26.183289 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:26.183265    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:26.184150 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:26.184134    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:11:26.406827 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:26.406776    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:27.330248 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:27.330209    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l" podUID="0e78d532-20ba-4881-801a-25d02b170bb0" containerName="payload-processing" containerID="cri-o://9b514da15517eb2e024b74b14bcf10e5189a552e06c675e149b6d5012c73afa3" gracePeriod=25
Oct 31 01:11:27.579946 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:27.579924    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"
Oct 31 01:11:27.681304 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:27.681238    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e78d532-20ba-4881-801a-25d02b170bb0-kube-api-access-9qddr\" (UniqueName: \"kubernetes.io/projected/0e78d532-20ba-4881-801a-25d02b170bb0-kube-api-access-9qddr\") pod \"0e78d532-20ba-4881-801a-25d02b170bb0\" (UID: \"0e78d532-20ba-4881-801a-25d02b170bb0\") "
Oct 31 01:11:27.681304 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:27.681300    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e78d532-20ba-4881-801a-25d02b170bb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78d532-20ba-4881-801a-25d02b170bb0-plugins-config-volume\") pod \"0e78d532-20ba-4881-801a-25d02b170bb0\" (UID: \"0e78d532-20ba-4881-801a-25d02b170bb0\") "
Oct 31 01:11:27.681682 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:27.681657    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e78d532-20ba-4881-801a-25d02b170bb0-plugins-config-volume" pod "0e78d532-20ba-4881-801a-25d02b170bb0" (UID: "0e78d532-20ba-4881-801a-25d02b170bb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:27.683389 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:27.683367    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e78d532-20ba-4881-801a-25d02b170bb0-kube-api-access-9qddr" pod "0e78d532-20ba-4881-801a-25d02b170bb0" (UID: "0e78d532-20ba-4881-801a-25d02b170bb0"). InnerVolumeSpecName "kube-api-access-9qddr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:27.782484 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:27.782447    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qddr\" (UniqueName: \"kubernetes.io/projected/0e78d532-20ba-4881-801a-25d02b170bb0-kube-api-access-9qddr\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:27.782484 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:27.782482    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78d532-20ba-4881-801a-25d02b170bb0-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:28.138501 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:28.138471    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:28.171263 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:28.171234    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 01:11:28.194484 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:28.194453    4476 generic.go:356] "Generic (PLEG): container finished" podID="0e78d532-20ba-4881-801a-25d02b170bb0" containerID="9b514da15517eb2e024b74b14bcf10e5189a552e06c675e149b6d5012c73afa3" exitCode=0
Oct 31 01:11:28.194612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:28.194495    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"
Oct 31 01:11:28.194612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:28.194539    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l" event={"ID":"0e78d532-20ba-4881-801a-25d02b170bb0","Type":"ContainerDied","Data":"9b514da15517eb2e024b74b14bcf10e5189a552e06c675e149b6d5012c73afa3"}
Oct 31 01:11:28.194612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:28.194582    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l" event={"ID":"0e78d532-20ba-4881-801a-25d02b170bb0","Type":"ContainerDied","Data":"ca3f0ec2bbc4b8f9089125adad0e3be9feca89b23c16c5de018c04f6bdcd9d57"}
Oct 31 01:11:28.194612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:28.194602    4476 scope.go:122] "RemoveContainer" containerID="9b514da15517eb2e024b74b14bcf10e5189a552e06c675e149b6d5012c73afa3"
Oct 31 01:11:28.204628 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:28.204614    4476 scope.go:122] "RemoveContainer" containerID="9b514da15517eb2e024b74b14bcf10e5189a552e06c675e149b6d5012c73afa3"
Oct 31 01:11:28.204885 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:28.204865    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b514da15517eb2e024b74b14bcf10e5189a552e06c675e149b6d5012c73afa3\": container with ID starting with 9b514da15517eb2e024b74b14bcf10e5189a552e06c675e149b6d5012c73afa3 not found: ID does not exist" containerID="9b514da15517eb2e024b74b14bcf10e5189a552e06c675e149b6d5012c73afa3"
Oct 31 01:11:28.204924 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:28.204894    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b514da15517eb2e024b74b14bcf10e5189a552e06c675e149b6d5012c73afa3"} err="failed to get container status \"9b514da15517eb2e024b74b14bcf10e5189a552e06c675e149b6d5012c73afa3\": rpc error: code = NotFound desc = could not find container \"9b514da15517eb2e024b74b14bcf10e5189a552e06c675e149b6d5012c73afa3\": container with ID starting with 9b514da15517eb2e024b74b14bcf10e5189a552e06c675e149b6d5012c73afa3 not found: ID does not exist"
Oct 31 01:11:28.337974 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:28.337945    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:28.397614 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:28.397534    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"]
Oct 31 01:11:28.401417 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:28.401382    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-935bafe7-fdd4b6f47-tfh7l"]
Oct 31 01:11:28.452904 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:28.452875    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e78d532-20ba-4881-801a-25d02b170bb0" path="/var/lib/kubelet/pods/0e78d532-20ba-4881-801a-25d02b170bb0/volumes"
Oct 31 01:11:28.535027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:28.535002    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:29.181127 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:29.181094    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:11:29.873944 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:29.873909    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:30.510849 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:30.510809    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:11:30.522702 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:30.522677    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:11:31.439735 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:31.439692    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:31.618677 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:31.618647    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:32.966795 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:32.966724    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:33.156098 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:33.156067    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:35.090138 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.090109    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"]
Oct 31 01:11:35.090647 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.090362    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx" podUID="507f992d-fb4a-44c0-aebc-de02ce8eb9f5" containerName="maas-api" containerID="cri-o://8263772b487f122dd70d4a2eade3434027a9ff7fff75d264147ba6cce1d548da" gracePeriod=30
Oct 31 01:11:35.236195 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.234063    4476 generic.go:356] "Generic (PLEG): container finished" podID="507f992d-fb4a-44c0-aebc-de02ce8eb9f5" containerID="8263772b487f122dd70d4a2eade3434027a9ff7fff75d264147ba6cce1d548da" exitCode=0
Oct 31 01:11:35.236195 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.234223    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx" event={"ID":"507f992d-fb4a-44c0-aebc-de02ce8eb9f5","Type":"ContainerDied","Data":"8263772b487f122dd70d4a2eade3434027a9ff7fff75d264147ba6cce1d548da"}
Oct 31 01:11:35.239455 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.239422    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:35.342272 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.342204    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:11:35.354379 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.354358    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-kube-api-access-96f89\" (UniqueName: \"kubernetes.io/projected/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-kube-api-access-96f89\") pod \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") "
Oct 31 01:11:35.354513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.354463    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-openshift-service-ca\") pod \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") "
Oct 31 01:11:35.354513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.354483    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-metrics-certs\") pod \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") "
Oct 31 01:11:35.354629 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.354519    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-maas-api-tls\") pod \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\" (UID: \"507f992d-fb4a-44c0-aebc-de02ce8eb9f5\") "
Oct 31 01:11:35.354886 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.354856    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-openshift-service-ca" pod "507f992d-fb4a-44c0-aebc-de02ce8eb9f5" (UID: "507f992d-fb4a-44c0-aebc-de02ce8eb9f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:35.356623 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.356594    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-kube-api-access-96f89" pod "507f992d-fb4a-44c0-aebc-de02ce8eb9f5" (UID: "507f992d-fb4a-44c0-aebc-de02ce8eb9f5"). InnerVolumeSpecName "kube-api-access-96f89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:35.356785 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.356759    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-metrics-certs" pod "507f992d-fb4a-44c0-aebc-de02ce8eb9f5" (UID: "507f992d-fb4a-44c0-aebc-de02ce8eb9f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:35.356854 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.356777    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-maas-api-tls" pod "507f992d-fb4a-44c0-aebc-de02ce8eb9f5" (UID: "507f992d-fb4a-44c0-aebc-de02ce8eb9f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:35.455711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.455679    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96f89\" (UniqueName: \"kubernetes.io/projected/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-kube-api-access-96f89\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:35.455711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.455708    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:35.455891 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.455718    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:35.455891 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.455728    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/507f992d-fb4a-44c0-aebc-de02ce8eb9f5-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:35.646632 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.646542    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:35.922430 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:35.922339    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:36.173476 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:36.173397    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:36.225845 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:36.225815    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"]
Oct 31 01:11:36.226123 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:36.226097    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" containerName="istio-proxy" containerID="cri-o://0ef38a4aa5e6ced5f61edb3a2f95f73d46434d6f8165d31834439793280a40f4" gracePeriod=30
Oct 31 01:11:36.240013 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:36.239978    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx" event={"ID":"507f992d-fb4a-44c0-aebc-de02ce8eb9f5","Type":"ContainerDied","Data":"c5808fdfc0316cc710634d1efbe6e376db7efccdcbff031394cee3c4d9df619c"}
Oct 31 01:11:36.240013 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:36.239999    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"
Oct 31 01:11:36.240219 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:36.240042    4476 scope.go:122] "RemoveContainer" containerID="8263772b487f122dd70d4a2eade3434027a9ff7fff75d264147ba6cce1d548da"
Oct 31 01:11:36.420141 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:36.420112    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:36.477914 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:36.477840    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"]
Oct 31 01:11:36.484290 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:36.484268    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-935bafe7-558cbbf94f-6mrtx"]
Oct 31 01:11:36.949216 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:36.949189    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:37.174062 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:37.174028    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:37.419579 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:37.419551    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:37.766666 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:37.766588    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:37.958216 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:37.958167    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:38.162891 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:38.162861    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:38.453670 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:38.453598    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="507f992d-fb4a-44c0-aebc-de02ce8eb9f5" path="/var/lib/kubelet/pods/507f992d-fb4a-44c0-aebc-de02ce8eb9f5/volumes"
Oct 31 01:11:38.521259 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:38.521162    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:39.145490 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:39.145458    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:39.345884 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:39.345850    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:40.196645 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:40.196613    4476 patch_prober.go:29] interesting pod/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:11:40.197018 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:40.196668    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:11:41.474002 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.473982    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:11:41.513821 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.513766    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e817033e-6407-4cea-88a3-ee4b801219b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e817033e-6407-4cea-88a3-ee4b801219b5-istio-podinfo\") pod \"e817033e-6407-4cea-88a3-ee4b801219b5\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") "
Oct 31 01:11:41.513821 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.513802    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e817033e-6407-4cea-88a3-ee4b801219b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e817033e-6407-4cea-88a3-ee4b801219b5-istiod-ca-cert\") pod \"e817033e-6407-4cea-88a3-ee4b801219b5\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") "
Oct 31 01:11:41.513990 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.513850    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-workload-certs\") pod \"e817033e-6407-4cea-88a3-ee4b801219b5\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") "
Oct 31 01:11:41.513990 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.513876    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-istio-envoy\") pod \"e817033e-6407-4cea-88a3-ee4b801219b5\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") "
Oct 31 01:11:41.513990 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.513925    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e817033e-6407-4cea-88a3-ee4b801219b5-kube-api-access-nffm4\" (UniqueName: \"kubernetes.io/projected/e817033e-6407-4cea-88a3-ee4b801219b5-kube-api-access-nffm4\") pod \"e817033e-6407-4cea-88a3-ee4b801219b5\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") "
Oct 31 01:11:41.513990 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.513951    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e817033e-6407-4cea-88a3-ee4b801219b5-istio-token\" (UniqueName: \"kubernetes.io/projected/e817033e-6407-4cea-88a3-ee4b801219b5-istio-token\") pod \"e817033e-6407-4cea-88a3-ee4b801219b5\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") "
Oct 31 01:11:41.514232 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.514001    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-credential-socket\") pod \"e817033e-6407-4cea-88a3-ee4b801219b5\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") "
Oct 31 01:11:41.514232 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.514051    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-workload-socket\") pod \"e817033e-6407-4cea-88a3-ee4b801219b5\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") "
Oct 31 01:11:41.514232 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.514116    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-istio-data\") pod \"e817033e-6407-4cea-88a3-ee4b801219b5\" (UID: \"e817033e-6407-4cea-88a3-ee4b801219b5\") "
Oct 31 01:11:41.514232 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.514142    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-workload-certs" pod "e817033e-6407-4cea-88a3-ee4b801219b5" (UID: "e817033e-6407-4cea-88a3-ee4b801219b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:41.514232 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.514211    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e817033e-6407-4cea-88a3-ee4b801219b5-istiod-ca-cert" pod "e817033e-6407-4cea-88a3-ee4b801219b5" (UID: "e817033e-6407-4cea-88a3-ee4b801219b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:41.514491 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.514396    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-credential-socket" pod "e817033e-6407-4cea-88a3-ee4b801219b5" (UID: "e817033e-6407-4cea-88a3-ee4b801219b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:41.514545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.514530    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-workload-socket" pod "e817033e-6407-4cea-88a3-ee4b801219b5" (UID: "e817033e-6407-4cea-88a3-ee4b801219b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:41.514597 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.514544    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-istio-data" pod "e817033e-6407-4cea-88a3-ee4b801219b5" (UID: "e817033e-6407-4cea-88a3-ee4b801219b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:41.514660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.514597    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:41.514660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.514618    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e817033e-6407-4cea-88a3-ee4b801219b5-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:41.514660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.514633    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:41.516249 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.516167    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-istio-envoy" pod "e817033e-6407-4cea-88a3-ee4b801219b5" (UID: "e817033e-6407-4cea-88a3-ee4b801219b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:41.516365 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.516252    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e817033e-6407-4cea-88a3-ee4b801219b5-kube-api-access-nffm4" pod "e817033e-6407-4cea-88a3-ee4b801219b5" (UID: "e817033e-6407-4cea-88a3-ee4b801219b5"). InnerVolumeSpecName "kube-api-access-nffm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:41.516583 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.516564    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e817033e-6407-4cea-88a3-ee4b801219b5-istio-podinfo" pod "e817033e-6407-4cea-88a3-ee4b801219b5" (UID: "e817033e-6407-4cea-88a3-ee4b801219b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:11:41.516630 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.516577    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e817033e-6407-4cea-88a3-ee4b801219b5-istio-token" pod "e817033e-6407-4cea-88a3-ee4b801219b5" (UID: "e817033e-6407-4cea-88a3-ee4b801219b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:41.615731 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.615709    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e817033e-6407-4cea-88a3-ee4b801219b5-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:41.615731 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.615731    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:41.615890 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.615741    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nffm4\" (UniqueName: \"kubernetes.io/projected/e817033e-6407-4cea-88a3-ee4b801219b5-kube-api-access-nffm4\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:41.615890 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.615751    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e817033e-6407-4cea-88a3-ee4b801219b5-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:41.615890 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.615760    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:41.615890 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.615768    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e817033e-6407-4cea-88a3-ee4b801219b5-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:41.690075 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.690039    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"]
Oct 31 01:11:41.690813 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.690319    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6" podUID="acb2731c-cb28-48f6-8848-34838bb47a95" containerName="maas-api" containerID="cri-o://6e8a9a73a3b830af3d696fc92a9b6c34f4b0174a578a07b9116be5a62cb2977f" gracePeriod=30
Oct 31 01:11:41.786516 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.786484    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:41.928522 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:41.928498    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:11:42.019665 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.019621    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-maas-api-tls\") pod \"acb2731c-cb28-48f6-8848-34838bb47a95\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") "
Oct 31 01:11:42.019829 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.019727    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb2731c-cb28-48f6-8848-34838bb47a95-kube-api-access-bsc7b\" (UniqueName: \"kubernetes.io/projected/acb2731c-cb28-48f6-8848-34838bb47a95-kube-api-access-bsc7b\") pod \"acb2731c-cb28-48f6-8848-34838bb47a95\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") "
Oct 31 01:11:42.019829 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.019785    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acb2731c-cb28-48f6-8848-34838bb47a95-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acb2731c-cb28-48f6-8848-34838bb47a95-openshift-service-ca\") pod \"acb2731c-cb28-48f6-8848-34838bb47a95\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") "
Oct 31 01:11:42.019960 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.019861    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-metrics-certs\" (UniqueName: \"kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-metrics-certs\") pod \"acb2731c-cb28-48f6-8848-34838bb47a95\" (UID: \"acb2731c-cb28-48f6-8848-34838bb47a95\") "
Oct 31 01:11:42.020605 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.020537    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb2731c-cb28-48f6-8848-34838bb47a95-openshift-service-ca" pod "acb2731c-cb28-48f6-8848-34838bb47a95" (UID: "acb2731c-cb28-48f6-8848-34838bb47a95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:42.022374 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.022344    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-metrics-certs" pod "acb2731c-cb28-48f6-8848-34838bb47a95" (UID: "acb2731c-cb28-48f6-8848-34838bb47a95"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:42.022507 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.022480    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-maas-api-tls" pod "acb2731c-cb28-48f6-8848-34838bb47a95" (UID: "acb2731c-cb28-48f6-8848-34838bb47a95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:42.022696 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.022670    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb2731c-cb28-48f6-8848-34838bb47a95-kube-api-access-bsc7b" pod "acb2731c-cb28-48f6-8848-34838bb47a95" (UID: "acb2731c-cb28-48f6-8848-34838bb47a95"). InnerVolumeSpecName "kube-api-access-bsc7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:42.121583 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.121513    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:42.121583 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.121536    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsc7b\" (UniqueName: \"kubernetes.io/projected/acb2731c-cb28-48f6-8848-34838bb47a95-kube-api-access-bsc7b\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:42.121583 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.121546    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acb2731c-cb28-48f6-8848-34838bb47a95-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:42.121583 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.121556    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acb2731c-cb28-48f6-8848-34838bb47a95-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:42.268893 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.268857    4476 generic.go:356] "Generic (PLEG): container finished" podID="e817033e-6407-4cea-88a3-ee4b801219b5" containerID="0ef38a4aa5e6ced5f61edb3a2f95f73d46434d6f8165d31834439793280a40f4" exitCode=0
Oct 31 01:11:42.269051 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.268897    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" event={"ID":"e817033e-6407-4cea-88a3-ee4b801219b5","Type":"ContainerDied","Data":"0ef38a4aa5e6ced5f61edb3a2f95f73d46434d6f8165d31834439793280a40f4"}
Oct 31 01:11:42.269051 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.268967    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk" event={"ID":"e817033e-6407-4cea-88a3-ee4b801219b5","Type":"ContainerDied","Data":"8b9851685b8e669de804ac3fd6067ad4ef6035df3eaf80cef189955398a9099e"}
Oct 31 01:11:42.269051 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.268998    4476 scope.go:122] "RemoveContainer" containerID="0ef38a4aa5e6ced5f61edb3a2f95f73d46434d6f8165d31834439793280a40f4"
Oct 31 01:11:42.269051 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.268914    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"
Oct 31 01:11:42.270447 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.270417    4476 generic.go:356] "Generic (PLEG): container finished" podID="acb2731c-cb28-48f6-8848-34838bb47a95" containerID="6e8a9a73a3b830af3d696fc92a9b6c34f4b0174a578a07b9116be5a62cb2977f" exitCode=0
Oct 31 01:11:42.270721 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.270464    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"
Oct 31 01:11:42.270721 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.270487    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6" event={"ID":"acb2731c-cb28-48f6-8848-34838bb47a95","Type":"ContainerDied","Data":"6e8a9a73a3b830af3d696fc92a9b6c34f4b0174a578a07b9116be5a62cb2977f"}
Oct 31 01:11:42.270721 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.270524    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6" event={"ID":"acb2731c-cb28-48f6-8848-34838bb47a95","Type":"ContainerDied","Data":"eed64e7e1d037f6a2d96a926a7852a5e655060d251b92843256b2fa169c07b41"}
Oct 31 01:11:42.277893 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.277871    4476 scope.go:122] "RemoveContainer" containerID="bf7da808748db6cfa121712ce4e7a9850532830c437aa47fc39bfc97e28764e5"
Oct 31 01:11:42.285845 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.285830    4476 scope.go:122] "RemoveContainer" containerID="0ef38a4aa5e6ced5f61edb3a2f95f73d46434d6f8165d31834439793280a40f4"
Oct 31 01:11:42.286110 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:42.286095    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef38a4aa5e6ced5f61edb3a2f95f73d46434d6f8165d31834439793280a40f4\": container with ID starting with 0ef38a4aa5e6ced5f61edb3a2f95f73d46434d6f8165d31834439793280a40f4 not found: ID does not exist" containerID="0ef38a4aa5e6ced5f61edb3a2f95f73d46434d6f8165d31834439793280a40f4"
Oct 31 01:11:42.286159 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.286117    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef38a4aa5e6ced5f61edb3a2f95f73d46434d6f8165d31834439793280a40f4"} err="failed to get container status \"0ef38a4aa5e6ced5f61edb3a2f95f73d46434d6f8165d31834439793280a40f4\": rpc error: code = NotFound desc = could not find container \"0ef38a4aa5e6ced5f61edb3a2f95f73d46434d6f8165d31834439793280a40f4\": container with ID starting with 0ef38a4aa5e6ced5f61edb3a2f95f73d46434d6f8165d31834439793280a40f4 not found: ID does not exist"
Oct 31 01:11:42.286159 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.286132    4476 scope.go:122] "RemoveContainer" containerID="bf7da808748db6cfa121712ce4e7a9850532830c437aa47fc39bfc97e28764e5"
Oct 31 01:11:42.286373 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:42.286355    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7da808748db6cfa121712ce4e7a9850532830c437aa47fc39bfc97e28764e5\": container with ID starting with bf7da808748db6cfa121712ce4e7a9850532830c437aa47fc39bfc97e28764e5 not found: ID does not exist" containerID="bf7da808748db6cfa121712ce4e7a9850532830c437aa47fc39bfc97e28764e5"
Oct 31 01:11:42.286439 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.286382    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7da808748db6cfa121712ce4e7a9850532830c437aa47fc39bfc97e28764e5"} err="failed to get container status \"bf7da808748db6cfa121712ce4e7a9850532830c437aa47fc39bfc97e28764e5\": rpc error: code = NotFound desc = could not find container \"bf7da808748db6cfa121712ce4e7a9850532830c437aa47fc39bfc97e28764e5\": container with ID starting with bf7da808748db6cfa121712ce4e7a9850532830c437aa47fc39bfc97e28764e5 not found: ID does not exist"
Oct 31 01:11:42.286439 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.286404    4476 scope.go:122] "RemoveContainer" containerID="6e8a9a73a3b830af3d696fc92a9b6c34f4b0174a578a07b9116be5a62cb2977f"
Oct 31 01:11:42.293927 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.293905    4476 scope.go:122] "RemoveContainer" containerID="6e8a9a73a3b830af3d696fc92a9b6c34f4b0174a578a07b9116be5a62cb2977f"
Oct 31 01:11:42.294187 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:42.294157    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8a9a73a3b830af3d696fc92a9b6c34f4b0174a578a07b9116be5a62cb2977f\": container with ID starting with 6e8a9a73a3b830af3d696fc92a9b6c34f4b0174a578a07b9116be5a62cb2977f not found: ID does not exist" containerID="6e8a9a73a3b830af3d696fc92a9b6c34f4b0174a578a07b9116be5a62cb2977f"
Oct 31 01:11:42.294262 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.294226    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8a9a73a3b830af3d696fc92a9b6c34f4b0174a578a07b9116be5a62cb2977f"} err="failed to get container status \"6e8a9a73a3b830af3d696fc92a9b6c34f4b0174a578a07b9116be5a62cb2977f\": rpc error: code = NotFound desc = could not find container \"6e8a9a73a3b830af3d696fc92a9b6c34f4b0174a578a07b9116be5a62cb2977f\": container with ID starting with 6e8a9a73a3b830af3d696fc92a9b6c34f4b0174a578a07b9116be5a62cb2977f not found: ID does not exist"
Oct 31 01:11:42.431111 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.431046    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"]
Oct 31 01:11:42.479989 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.479966    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-79f7219a-openshift-default-d9947f8b9-qsvvk"]
Oct 31 01:11:42.542719 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.542696    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"]
Oct 31 01:11:42.546106 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:42.546085    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-60c85e15-674d995f8c-mbvq6"]
Oct 31 01:11:43.230347 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.230314    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"]
Oct 31 01:11:43.230678 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.230667    4476 cpu_manager.go:414] "containerMap: removing container" podUID="acb2731c-cb28-48f6-8848-34838bb47a95" containerName="maas-api"
Oct 31 01:11:43.230717 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.230679    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb2731c-cb28-48f6-8848-34838bb47a95" containerName="maas-api"
Oct 31 01:11:43.230717 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.230697    4476 cpu_manager.go:414] "containerMap: removing container" podUID="0e78d532-20ba-4881-801a-25d02b170bb0" containerName="payload-processing"
Oct 31 01:11:43.230717 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.230702    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e78d532-20ba-4881-801a-25d02b170bb0" containerName="payload-processing"
Oct 31 01:11:43.230717 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.230713    4476 cpu_manager.go:414] "containerMap: removing container" podUID="507f992d-fb4a-44c0-aebc-de02ce8eb9f5" containerName="maas-api"
Oct 31 01:11:43.230717 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.230718    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507f992d-fb4a-44c0-aebc-de02ce8eb9f5" containerName="maas-api"
Oct 31 01:11:43.230870 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.230731    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" containerName="istio-proxy"
Oct 31 01:11:43.230870 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.230735    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" containerName="istio-proxy"
Oct 31 01:11:43.230870 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.230785    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e78d532-20ba-4881-801a-25d02b170bb0" containerName="payload-processing"
Oct 31 01:11:43.230870 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.230795    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="507f992d-fb4a-44c0-aebc-de02ce8eb9f5" containerName="maas-api"
Oct 31 01:11:43.230870 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.230803    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" containerName="istio-proxy"
Oct 31 01:11:43.230870 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.230809    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" containerName="istio-proxy"
Oct 31 01:11:43.230870 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.230816    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb2731c-cb28-48f6-8848-34838bb47a95" containerName="maas-api"
Oct 31 01:11:43.233682 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.233659    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:43.285383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.285359    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-acd385-serving-cert\""
Oct 31 01:11:43.302867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.302844    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"]
Oct 31 01:11:43.320335 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.320310    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:43.333513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.333485    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b467a1-84b0-4f07-acda-74bf7ccb1be2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-acd385-79d758567-h2n7c\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:43.333641 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.333528    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-acd385-79d758567-h2n7c\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:43.333641 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.333566    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-metrics-certs\") pod \"maas-api-e2e-worker-w4-acd385-79d758567-h2n7c\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:43.333766 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.333640    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcsm7\" (UniqueName: \"kubernetes.io/projected/66b467a1-84b0-4f07-acda-74bf7ccb1be2-kube-api-access-gcsm7\") pod \"maas-api-e2e-worker-w4-acd385-79d758567-h2n7c\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:43.434906 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.434874    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b467a1-84b0-4f07-acda-74bf7ccb1be2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-acd385-79d758567-h2n7c\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:43.435085 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.434918    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-acd385-79d758567-h2n7c\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:43.435085 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.434961    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-metrics-certs\") pod \"maas-api-e2e-worker-w4-acd385-79d758567-h2n7c\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:43.435085 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.434985    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcsm7\" (UniqueName: \"kubernetes.io/projected/66b467a1-84b0-4f07-acda-74bf7ccb1be2-kube-api-access-gcsm7\") pod \"maas-api-e2e-worker-w4-acd385-79d758567-h2n7c\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:43.435299 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:43.435112    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-serving-cert: secret "maas-api-e2e-worker-w4-acd385-serving-cert" not found
Oct 31 01:11:43.435299 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:43.435204    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-maas-api-tls podName:66b467a1-84b0-4f07-acda-74bf7ccb1be2 nodeName:}" failed. No retries permitted until 2026-10-31 01:11:43.93516809 +0000 UTC m=+913.625051214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-maas-api-tls") pod "maas-api-e2e-worker-w4-acd385-79d758567-h2n7c" (UID: "66b467a1-84b0-4f07-acda-74bf7ccb1be2") : secret "maas-api-e2e-worker-w4-acd385-serving-cert" not found
Oct 31 01:11:43.435679 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.435660    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b467a1-84b0-4f07-acda-74bf7ccb1be2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-acd385-79d758567-h2n7c\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:43.437777 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.437751    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-metrics-certs\") pod \"maas-api-e2e-worker-w4-acd385-79d758567-h2n7c\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:43.491630 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.491568    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcsm7\" (UniqueName: \"kubernetes.io/projected/66b467a1-84b0-4f07-acda-74bf7ccb1be2-kube-api-access-gcsm7\") pod \"maas-api-e2e-worker-w4-acd385-79d758567-h2n7c\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:43.507604 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.507579    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:43.692082 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.692056    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:43.873200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.873151    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:43.940714 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.940676    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-acd385-79d758567-h2n7c\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:43.943211 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:43.943195    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-acd385-79d758567-h2n7c\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:44.146794 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:44.146719    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:44.397418 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:44.397329    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:44.452723 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:44.452692    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acb2731c-cb28-48f6-8848-34838bb47a95" path="/var/lib/kubelet/pods/acb2731c-cb28-48f6-8848-34838bb47a95/volumes"
Oct 31 01:11:44.453351 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:44.453328    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" path="/var/lib/kubelet/pods/e817033e-6407-4cea-88a3-ee4b801219b5/volumes"
Oct 31 01:11:44.518718 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:44.518698    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"]
Oct 31 01:11:44.520793 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:11:44.520768    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b467a1_84b0_4f07_acda_74bf7ccb1be2.slice/crio-e783ee9a0de949d5bce153168d5bb065bb4e1bc61a20b9e5631b1af70b3be1e3 WatchSource:0}: Error finding container e783ee9a0de949d5bce153168d5bb065bb4e1bc61a20b9e5631b1af70b3be1e3: Status 404 returned error can't find the container with id e783ee9a0de949d5bce153168d5bb065bb4e1bc61a20b9e5631b1af70b3be1e3
Oct 31 01:11:44.988674 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:44.988640    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:45.181513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:45.181464    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:45.286916 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:45.286883    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c" event={"ID":"66b467a1-84b0-4f07-acda-74bf7ccb1be2","Type":"ContainerStarted","Data":"e783ee9a0de949d5bce153168d5bb065bb4e1bc61a20b9e5631b1af70b3be1e3"}
Oct 31 01:11:45.842975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:45.842947    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:46.293383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:46.293352    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c" event={"ID":"66b467a1-84b0-4f07-acda-74bf7ccb1be2","Type":"ContainerStarted","Data":"344ce5a99fe87c08ecfb21b8b79162a980f09e5c04e369df05eabdc7c2e58187"}
Oct 31 01:11:46.293588 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:46.293436    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:46.412349 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:46.412292    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c" podStartSLOduration=2.591775729 podStartE2EDuration="3.412278114s" totalImagesPullingTime="820.502385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:46.411785736 +0000 UTC m=+916.101668866" watchObservedRunningTime="2026-10-31 01:11:46.412278114 +0000 UTC m=+916.102161276"
Oct 31 01:11:48.885832 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:48.885801    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:49.481199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.468531    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"]
Oct 31 01:11:49.482077 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.482046    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" containerName="istio-proxy"
Oct 31 01:11:49.482267 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.482251    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e817033e-6407-4cea-88a3-ee4b801219b5" containerName="istio-proxy"
Oct 31 01:11:49.485960 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.485939    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.550421 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.550393    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-26fbc4\"/\"default-dockercfg-jd84n\""
Oct 31 01:11:49.586928 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.586891    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-26fbc4\"/\"openshift-service-ca.crt\""
Oct 31 01:11:49.587092 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.586930    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-26fbc4\"/\"kube-root-ca.crt\""
Oct 31 01:11:49.587092 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.586997    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-26fbc4\"/\"simulator-w0-26fbc4-kserve-self-signed-certs\""
Oct 31 01:11:49.588981 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.588960    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"]
Oct 31 01:11:49.592903 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.592883    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:49.597011 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.596991    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7db\" (UniqueName: \"kubernetes.io/projected/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-kube-api-access-jl7db\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.597067 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.597026    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-tmp-dir\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.597067 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.597047    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-dshm\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.597318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.597160    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-tls-certs\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.597318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.597207    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-home\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.597318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.597298    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-model-cache\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.698762 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.698729    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-dshm\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.698762 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.698778    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-tls-certs\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.699023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.698798    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-home\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.699023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.698849    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-model-cache\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.699023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.698882    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7db\" (UniqueName: \"kubernetes.io/projected/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-kube-api-access-jl7db\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.699023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.698906    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-tmp-dir\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.699325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.699297    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-home\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.699482 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.699329    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-model-cache\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.699482 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.699357    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-tmp-dir\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.701329 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.701311    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-dshm\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.701493 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.701478    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-tls-certs\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.757910 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.757852    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7db\" (UniqueName: \"kubernetes.io/projected/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-kube-api-access-jl7db\") pod \"simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") " pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:49.799473 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:49.799444    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:50.084207 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.080943    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:50.175975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.175566    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"]
Oct 31 01:11:50.191211 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:11:50.190281    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfffaf3f9_2e8d_4aa1_a5d1_c3db5f0f5194.slice/crio-ccb3ec40049a5652ef1f827762a84a43f2316b4b09db06179ebebaaf702da87a WatchSource:0}: Error finding container ccb3ec40049a5652ef1f827762a84a43f2316b4b09db06179ebebaaf702da87a: Status 404 returned error can't find the container with id ccb3ec40049a5652ef1f827762a84a43f2316b4b09db06179ebebaaf702da87a
Oct 31 01:11:50.284199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.283896    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:50.320645 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.320609    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp" event={"ID":"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194","Type":"ContainerStarted","Data":"ed25902217710808e743d699389cadc3594b3882090ff1a2dc9d359d089e838b"}
Oct 31 01:11:50.320645 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.320650    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp" event={"ID":"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194","Type":"ContainerStarted","Data":"ccb3ec40049a5652ef1f827762a84a43f2316b4b09db06179ebebaaf702da87a"}
Oct 31 01:11:50.438702 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.438613    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp" podStartSLOduration=1.4386009 podStartE2EDuration="1.4386009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:50.435789852 +0000 UTC m=+920.125672992" watchObservedRunningTime="2026-10-31 01:11:50.4386009 +0000 UTC m=+920.128484027"
Oct 31 01:11:50.481201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.480573    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"]
Oct 31 01:11:50.487202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.487163    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"
Oct 31 01:11:50.506119 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.506095    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct\" (UID: \"2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"
Oct 31 01:11:50.506240 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.506133    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r2m2\" (UniqueName: \"kubernetes.io/projected/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912-kube-api-access-8r2m2\") pod \"payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct\" (UID: \"2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"
Oct 31 01:11:50.538793 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.538771    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e621db-dockercfg-pm6fd\""
Oct 31 01:11:50.538879 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.538792    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e621db\""
Oct 31 01:11:50.547403 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.547384    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"]
Oct 31 01:11:50.605522 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.605485    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"]
Oct 31 01:11:50.606915 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.606885    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct\" (UID: \"2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"
Oct 31 01:11:50.607043 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.606927    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r2m2\" (UniqueName: \"kubernetes.io/projected/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912-kube-api-access-8r2m2\") pod \"payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct\" (UID: \"2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"
Oct 31 01:11:50.607723 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.607701    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct\" (UID: \"2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"
Oct 31 01:11:50.609009 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.608994    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"
Oct 31 01:11:50.687330 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.687294    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r2m2\" (UniqueName: \"kubernetes.io/projected/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912-kube-api-access-8r2m2\") pod \"payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct\" (UID: \"2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"
Oct 31 01:11:50.688206 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.688159    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"]
Oct 31 01:11:50.707876 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.707824    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vglcd\" (UniqueName: \"kubernetes.io/projected/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca-kube-api-access-vglcd\") pod \"payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn\" (UID: \"a7b29e19-6e82-4c47-a9f6-d66f60afd3ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"
Oct 31 01:11:50.708130 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.708115    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn\" (UID: \"a7b29e19-6e82-4c47-a9f6-d66f60afd3ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"
Oct 31 01:11:50.799915 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.799883    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:50.800097 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.799962    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"
Oct 31 01:11:50.809599 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.809467    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn\" (UID: \"a7b29e19-6e82-4c47-a9f6-d66f60afd3ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"
Oct 31 01:11:50.809599 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.809586    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vglcd\" (UniqueName: \"kubernetes.io/projected/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca-kube-api-access-vglcd\") pod \"payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn\" (UID: \"a7b29e19-6e82-4c47-a9f6-d66f60afd3ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"
Oct 31 01:11:50.810397 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.810373    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn\" (UID: \"a7b29e19-6e82-4c47-a9f6-d66f60afd3ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"
Oct 31 01:11:50.894145 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.894110    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vglcd\" (UniqueName: \"kubernetes.io/projected/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca-kube-api-access-vglcd\") pod \"payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn\" (UID: \"a7b29e19-6e82-4c47-a9f6-d66f60afd3ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"
Oct 31 01:11:50.921294 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:50.921266    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"
Oct 31 01:11:51.183203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:51.175635    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"]
Oct 31 01:11:51.188209 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:11:51.188155    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc73dbe_90f0_4dc0_9ce9_c1e5cec12912.slice/crio-ee056a32cfc66d9eec37f7af2669c909cd77988ae6bffe7c43060bd944b2f7c2 WatchSource:0}: Error finding container ee056a32cfc66d9eec37f7af2669c909cd77988ae6bffe7c43060bd944b2f7c2: Status 404 returned error can't find the container with id ee056a32cfc66d9eec37f7af2669c909cd77988ae6bffe7c43060bd944b2f7c2
Oct 31 01:11:51.316055 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:51.316033    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"]
Oct 31 01:11:51.318584 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:11:51.318557    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b29e19_6e82_4c47_a9f6_d66f60afd3ca.slice/crio-37478683f9fcdb1e2bea46d6f82f4ade89adade9d476c2cc855f3310451558e0 WatchSource:0}: Error finding container 37478683f9fcdb1e2bea46d6f82f4ade89adade9d476c2cc855f3310451558e0: Status 404 returned error can't find the container with id 37478683f9fcdb1e2bea46d6f82f4ade89adade9d476c2cc855f3310451558e0
Oct 31 01:11:51.329941 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:51.329909    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn" event={"ID":"a7b29e19-6e82-4c47-a9f6-d66f60afd3ca","Type":"ContainerStarted","Data":"37478683f9fcdb1e2bea46d6f82f4ade89adade9d476c2cc855f3310451558e0"}
Oct 31 01:11:51.331498 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:51.331475    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct" event={"ID":"2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912","Type":"ContainerStarted","Data":"874e7836b191eeb22b86ad6b2d42ddab91f5491b571f438190da765ff095ec7e"}
Oct 31 01:11:51.331593 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:51.331513    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct" event={"ID":"2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912","Type":"ContainerStarted","Data":"ee056a32cfc66d9eec37f7af2669c909cd77988ae6bffe7c43060bd944b2f7c2"}
Oct 31 01:11:51.445810 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:51.445762    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct" podStartSLOduration=1.445748868 podStartE2EDuration="1.445748868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:51.444976934 +0000 UTC m=+921.134860064" watchObservedRunningTime="2026-10-31 01:11:51.445748868 +0000 UTC m=+921.135631994"
Oct 31 01:11:52.026821 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:52.026783    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:52.303784 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:52.303759    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 01:11:52.304220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:52.304170    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:52.337239 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:52.337208    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn" event={"ID":"a7b29e19-6e82-4c47-a9f6-d66f60afd3ca","Type":"ContainerStarted","Data":"c246c9b74ff834a5cf1711b363d84d70425cc6c1b7c0a316a249d4c7efd8a808"}
Oct 31 01:11:52.337744 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:52.337389    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"
Oct 31 01:11:52.337744 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:52.337427    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"
Oct 31 01:11:52.498738 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:52.498452    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:52.555253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:52.555137    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn" podStartSLOduration=2.555123997 podStartE2EDuration="2.555123997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:52.553753069 +0000 UTC m=+922.243636197" watchObservedRunningTime="2026-10-31 01:11:52.555123997 +0000 UTC m=+922.245007158"
Oct 31 01:11:54.010979 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.010946    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:54.136023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.135989    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"]
Oct 31 01:11:54.139754 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.139738    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.191633 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.191603    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:54.200226 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.200202    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"]
Oct 31 01:11:54.245433 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.245390    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.245572 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.245480    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh4lp\" (UniqueName: \"kubernetes.io/projected/b5b406da-fb74-4089-914d-569cc542526b-kube-api-access-fh4lp\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.245625 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.245598    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b406da-fb74-4089-914d-569cc542526b-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.245662 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.245623    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b406da-fb74-4089-914d-569cc542526b-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.245662 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.245646    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.245720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.245671    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.245757 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.245710    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.245798 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.245783    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.245847 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.245837    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b406da-fb74-4089-914d-569cc542526b-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.347034 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.347005    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.347233 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.347058    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b406da-fb74-4089-914d-569cc542526b-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.347233 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.347089    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.347233 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.347105    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh4lp\" (UniqueName: \"kubernetes.io/projected/b5b406da-fb74-4089-914d-569cc542526b-kube-api-access-fh4lp\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.347233 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.347154    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b406da-fb74-4089-914d-569cc542526b-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.347233 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.347191    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b406da-fb74-4089-914d-569cc542526b-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.347233 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.347217    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.347603 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.347254    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.347603 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.347277    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.347603 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.347571    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.347748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.347679    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.347825 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.347807    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.348160 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.348133    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b406da-fb74-4089-914d-569cc542526b-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.348463 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.348429    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.349625 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.349596    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.349827 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.349808    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b406da-fb74-4089-914d-569cc542526b-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.403316 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.403295    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b406da-fb74-4089-914d-569cc542526b-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.403798 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.403775    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh4lp\" (UniqueName: \"kubernetes.io/projected/b5b406da-fb74-4089-914d-569cc542526b-kube-api-access-fh4lp\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.452541 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.452516    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:54.897877 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.897850    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"]
Oct 31 01:11:54.899471 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:11:54.899435    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b406da_fb74_4089_914d_569cc542526b.slice/crio-f993665fca3de5bdacbd55066c943d57029cf33f7d7227a67816b3547d574709 WatchSource:0}: Error finding container f993665fca3de5bdacbd55066c943d57029cf33f7d7227a67816b3547d574709: Status 404 returned error can't find the container with id f993665fca3de5bdacbd55066c943d57029cf33f7d7227a67816b3547d574709
Oct 31 01:11:54.902079 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.902044    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:54.902193 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.902111    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:54.902193 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.902143    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:54.921789 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:54.921764    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:55.332026 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.331991    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"]
Oct 31 01:11:55.336315 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.336294    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.362478 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.356217    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk" event={"ID":"b5b406da-fb74-4089-914d-569cc542526b","Type":"ContainerStarted","Data":"0351f49235332d2410549ea6d305286b5639ce383d9330aeb2dca4d269e23353"}
Oct 31 01:11:55.362478 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.356286    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk" event={"ID":"b5b406da-fb74-4089-914d-569cc542526b","Type":"ContainerStarted","Data":"f993665fca3de5bdacbd55066c943d57029cf33f7d7227a67816b3547d574709"}
Oct 31 01:11:55.390212 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.388520    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ec695b-openshift-default-dockercfg-ksrrv\""
Oct 31 01:11:55.450803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.450763    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"]
Oct 31 01:11:55.453122 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.453067    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:11:55.454376 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.454353    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:11:55.454498 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.454398    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk" podUID="b5b406da-fb74-4089-914d-569cc542526b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 01:11:55.458147 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.458128    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.458244 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.458162    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.458329 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.458306    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.458393 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.458374    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.458453 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.458411    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.458453 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.458437    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.458623 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.458603    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8tfk\" (UniqueName: \"kubernetes.io/projected/7c71078d-3e0f-4e16-9df8-7cabafce5af9-kube-api-access-x8tfk\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.458692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.458644    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.458692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.458674    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.472216 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.469640    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:55.588260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.568770    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.588574 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.588528    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.588684 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.588635    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.588755 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.588722    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.588812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.588751    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.588812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.588781    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.588913 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.588823    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8tfk\" (UniqueName: \"kubernetes.io/projected/7c71078d-3e0f-4e16-9df8-7cabafce5af9-kube-api-access-x8tfk\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.588913 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.588857    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.588913 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.588890    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.589336 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.580010    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.590000 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.589973    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.590695 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.590669    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.590819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.590776    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.591030 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.591004    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.593124 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.593099    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.593237 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.593201    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.648694 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.648655    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.649028 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.649009    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8tfk\" (UniqueName: \"kubernetes.io/projected/7c71078d-3e0f-4e16-9df8-7cabafce5af9-kube-api-access-x8tfk\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.650755 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.650723    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:55.684443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:55.684355    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:56.093637 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:56.093568    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk" podStartSLOduration=2.093549411 podStartE2EDuration="2.093549411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:55.590332524 +0000 UTC m=+925.280215669" watchObservedRunningTime="2026-10-31 01:11:56.093549411 +0000 UTC m=+925.783432539"
Oct 31 01:11:56.095896 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:56.095872    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"]
Oct 31 01:11:56.136750 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:11:56.136721    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c71078d_3e0f_4e16_9df8_7cabafce5af9.slice/crio-9f6199826e7b3a5306015fb73c803c244730ce109d0fa24eae438cfb576cf5e7 WatchSource:0}: Error finding container 9f6199826e7b3a5306015fb73c803c244730ce109d0fa24eae438cfb576cf5e7: Status 404 returned error can't find the container with id 9f6199826e7b3a5306015fb73c803c244730ce109d0fa24eae438cfb576cf5e7
Oct 31 01:11:56.138839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:56.138808    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:56.138891 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:56.138876    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:56.138923 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:56.138905    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:56.386470 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:56.386284    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" event={"ID":"7c71078d-3e0f-4e16-9df8-7cabafce5af9","Type":"ContainerStarted","Data":"57c12101cd32a0a9e60b93f3e966b3324e78680563b69db77d9ea95543da473f"}
Oct 31 01:11:56.386470 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:56.386360    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" event={"ID":"7c71078d-3e0f-4e16-9df8-7cabafce5af9","Type":"ContainerStarted","Data":"9f6199826e7b3a5306015fb73c803c244730ce109d0fa24eae438cfb576cf5e7"}
Oct 31 01:11:56.453075 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:56.453041    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:11:56.453263 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:56.453102    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk" podUID="b5b406da-fb74-4089-914d-569cc542526b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 01:11:56.503929 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:56.503880    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" podStartSLOduration=1.503866077 podStartE2EDuration="1.503866077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:56.501816358 +0000 UTC m=+926.191699507" watchObservedRunningTime="2026-10-31 01:11:56.503866077 +0000 UTC m=+926.193749261"
Oct 31 01:11:56.651111 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:56.651038    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:57.232318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:57.232280    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:11:57.232491 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:57.232353    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:11:57.284999 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:57.284955    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:57.453846 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:57.453813    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:11:57.454279 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:57.453875    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk" podUID="b5b406da-fb74-4089-914d-569cc542526b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 01:11:57.488135 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:57.488021    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:57.657236 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:57.657207    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:58.405259 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:58.405214    4476 generic.go:356] "Generic (PLEG): container finished" podID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerID="98b36cc107b9f5365f69d25639d4932f3ec13182c2273fb057ff950bedea398c" exitCode=0
Oct 31 01:11:58.405473 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:58.405269    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" event={"ID":"43eb8302-cba6-4f09-a38c-d548c01abb46","Type":"ContainerDied","Data":"98b36cc107b9f5365f69d25639d4932f3ec13182c2273fb057ff950bedea398c"}
Oct 31 01:11:58.405717 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:58.405701    4476 scope.go:122] "RemoveContainer" containerID="98b36cc107b9f5365f69d25639d4932f3ec13182c2273fb057ff950bedea398c"
Oct 31 01:11:58.405844 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:58.405831    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:58.406937 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:58.406920    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:11:58.407058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:58.406953    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:58.407058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:58.407038    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:58.407169 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:58.407085    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:58.453253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:58.453215    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:11:58.453411 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:58.453268    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk" podUID="b5b406da-fb74-4089-914d-569cc542526b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 01:11:58.929799 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:58.929767    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"]
Oct 31 01:11:58.943915 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:58.943886    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:58.985204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:58.984786    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"]
Oct 31 01:11:58.997208 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:58.996642    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-e621db-serving-cert\""
Oct 31 01:11:59.044775 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.044739    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-metrics-certs\") pod \"maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:59.044963 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.044806    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:59.044963 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.044929    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:59.045067 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.044999    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prkcf\" (UniqueName: \"kubernetes.io/projected/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-kube-api-access-prkcf\") pod \"maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:59.145704 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.145663    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:59.145903 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.145744    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prkcf\" (UniqueName: \"kubernetes.io/projected/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-kube-api-access-prkcf\") pod \"maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:59.145903 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.145818    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-metrics-certs\") pod \"maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:59.145903 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.145860    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:59.146494 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:59.146300    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-serving-cert: secret "maas-api-e2e-worker-w2-e621db-serving-cert" not found
Oct 31 01:11:59.146494 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:59.146357    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-maas-api-tls podName:5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5 nodeName:}" failed. No retries permitted until 2026-10-31 01:11:59.646341969 +0000 UTC m=+929.336225076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-maas-api-tls") pod "maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9" (UID: "5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5") : secret "maas-api-e2e-worker-w2-e621db-serving-cert" not found
Oct 31 01:11:59.147300 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.147252    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:59.149397 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.149374    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-metrics-certs\") pod \"maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:59.149495 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.149423    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:59.220938 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.220824    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prkcf\" (UniqueName: \"kubernetes.io/projected/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-kube-api-access-prkcf\") pod \"maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:59.247271 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:11:59.247237    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:11:59.334320 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.334291    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:59.418773 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.418678    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" event={"ID":"43eb8302-cba6-4f09-a38c-d548c01abb46","Type":"ContainerStarted","Data":"65f5dbd709cf85fdc89e1a38ff2155b466d266d30e05a40a5b627309c8074f5e"}
Oct 31 01:11:59.453276 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.453248    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:11:59.453508 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.453477    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk" podUID="b5b406da-fb74-4089-914d-569cc542526b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 01:11:59.515694 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.515611    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:59.651826 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.651794    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:59.654545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.654512    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:59.711857 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.711820    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:59.804211 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.803505    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:59.822268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.822244    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:11:59.904211 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.903587    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:11:59.918363 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:11:59.918335    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:00.355291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:00.355270    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"]
Oct 31 01:12:00.357442 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:12:00.357400    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9d6d08_fc6a_4c3d_8ae3_6bed8194d2b5.slice/crio-544b534580dd41bfb0f3d6a87364d2c738f97451858528a81501211dfd75e4d2 WatchSource:0}: Error finding container 544b534580dd41bfb0f3d6a87364d2c738f97451858528a81501211dfd75e4d2: Status 404 returned error can't find the container with id 544b534580dd41bfb0f3d6a87364d2c738f97451858528a81501211dfd75e4d2
Oct 31 01:12:00.358810 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:00.358790    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:12:00.398200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:00.394398    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:12:00.398200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:00.394905    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 01:12:00.398200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:00.394972    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 01:12:00.424030 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:00.423998    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9" event={"ID":"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5","Type":"ContainerStarted","Data":"544b534580dd41bfb0f3d6a87364d2c738f97451858528a81501211dfd75e4d2"}
Oct 31 01:12:00.435984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:00.435934    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:12:00.453084 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:00.453063    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:12:00.453227 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:00.453104    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk" podUID="b5b406da-fb74-4089-914d-569cc542526b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 01:12:01.394759 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:01.394719    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 01:12:01.395278 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:01.394780    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 01:12:01.430872 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:01.430839    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9" event={"ID":"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5","Type":"ContainerStarted","Data":"2d8b5e6dcd8579ec079396d1d16cd4fd2ea6dc4614b95c5c0a8784cac0713246"}
Oct 31 01:12:01.431021 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:01.430915    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:12:01.456840 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:01.456777    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:12:01.457080 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:01.457063    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:12:01.457770 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:01.457753    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 01:12:01.553678 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:01.553623    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9" podStartSLOduration=2.711644985 podStartE2EDuration="3.553606129s" totalImagesPullingTime="841.961144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:01.551902322 +0000 UTC m=+931.241785461" watchObservedRunningTime="2026-10-31 01:12:01.553606129 +0000 UTC m=+931.243489256"
Oct 31 01:12:02.394676 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:02.394641    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 01:12:02.394867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:02.394695    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 01:12:03.345476 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:03.345449    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"
Oct 31 01:12:03.345649 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:03.345597    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"
Oct 31 01:12:03.393757 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:03.393721    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 01:12:03.393922 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:03.393765    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 01:12:04.393739 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:04.393703    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 01:12:04.394137 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:04.393752    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 01:12:04.599063 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:04.599035    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:04.783048 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:04.783021    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:04.966046 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:04.966014    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:05.398037 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:05.398007    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:12:05.398424 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:05.398364    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:12:05.399049 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:05.399032    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:12:07.438312 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:07.438280    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9" podUID="5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": dial tcp 10.133.0.59:8443: connect: connection refused"
Oct 31 01:12:09.298148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:09.298111    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:11.481856 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:11.481822    4476 generic.go:356] "Generic (PLEG): container finished" podID="5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" containerID="2d8b5e6dcd8579ec079396d1d16cd4fd2ea6dc4614b95c5c0a8784cac0713246" exitCode=1
Oct 31 01:12:11.482234 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:11.481879    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9" event={"ID":"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5","Type":"ContainerDied","Data":"2d8b5e6dcd8579ec079396d1d16cd4fd2ea6dc4614b95c5c0a8784cac0713246"}
Oct 31 01:12:11.482295 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:11.482237    4476 scope.go:122] "RemoveContainer" containerID="2d8b5e6dcd8579ec079396d1d16cd4fd2ea6dc4614b95c5c0a8784cac0713246"
Oct 31 01:12:12.488820 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:12.488786    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9" event={"ID":"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5","Type":"ContainerStarted","Data":"64d4df4ff64bbd979fa9e986ce20dd75ecd98372346c4bad20666484b34cb218"}
Oct 31 01:12:12.489208 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:12.488990    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:12:13.545414 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:13.545373    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:14.134806 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.134778    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:14.317413 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.317382    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:14.465366 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.465295    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"]
Oct 31 01:12:14.468964 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.468949    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:14.520645 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.520621    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-97b317-serving-cert\""
Oct 31 01:12:14.572606 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.572582    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"]
Oct 31 01:12:14.636007 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.635980    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3734f793-a26d-4f0b-8f5a-954b10bb51c7-metrics-certs\") pod \"maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:14.636157 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.636022    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3734f793-a26d-4f0b-8f5a-954b10bb51c7-maas-api-tls\") pod \"maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:14.636157 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.636041    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3734f793-a26d-4f0b-8f5a-954b10bb51c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:14.636157 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.636146    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pskbn\" (UniqueName: \"kubernetes.io/projected/3734f793-a26d-4f0b-8f5a-954b10bb51c7-kube-api-access-pskbn\") pod \"maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:14.710165 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.710136    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:14.737169 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.737095    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3734f793-a26d-4f0b-8f5a-954b10bb51c7-metrics-certs\") pod \"maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:14.737169 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.737157    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3734f793-a26d-4f0b-8f5a-954b10bb51c7-maas-api-tls\") pod \"maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:14.737405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.737200    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3734f793-a26d-4f0b-8f5a-954b10bb51c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:14.737405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.737282    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pskbn\" (UniqueName: \"kubernetes.io/projected/3734f793-a26d-4f0b-8f5a-954b10bb51c7-kube-api-access-pskbn\") pod \"maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:14.737858 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.737835    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3734f793-a26d-4f0b-8f5a-954b10bb51c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:14.739709 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.739691    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3734f793-a26d-4f0b-8f5a-954b10bb51c7-metrics-certs\") pod \"maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:14.739812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.739739    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3734f793-a26d-4f0b-8f5a-954b10bb51c7-maas-api-tls\") pod \"maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:14.794575 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.794543    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pskbn\" (UniqueName: \"kubernetes.io/projected/3734f793-a26d-4f0b-8f5a-954b10bb51c7-kube-api-access-pskbn\") pod \"maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:14.886822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:14.886790    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:15.081221 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:15.081193    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:15.457029 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:15.457002    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"]
Oct 31 01:12:15.458764 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:12:15.458741    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3734f793_a26d_4f0b_8f5a_954b10bb51c7.slice/crio-d605686330f1b4f8007dc91235a3a8226c6af26ba2088d017f818e47ad4309e7 WatchSource:0}: Error finding container d605686330f1b4f8007dc91235a3a8226c6af26ba2088d017f818e47ad4309e7: Status 404 returned error can't find the container with id d605686330f1b4f8007dc91235a3a8226c6af26ba2088d017f818e47ad4309e7
Oct 31 01:12:15.504906 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:15.504877    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5" event={"ID":"3734f793-a26d-4f0b-8f5a-954b10bb51c7","Type":"ContainerStarted","Data":"d605686330f1b4f8007dc91235a3a8226c6af26ba2088d017f818e47ad4309e7"}
Oct 31 01:12:15.784984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:15.784951    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:16.511685 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:16.511647    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5" event={"ID":"3734f793-a26d-4f0b-8f5a-954b10bb51c7","Type":"ContainerStarted","Data":"5948020fbe3df2043cf240ecf4e4b3eb8a581e26033a41e4c347ce61cc2ade19"}
Oct 31 01:12:16.511878 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:16.511778    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:16.632094 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:16.632002    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5" podStartSLOduration=1.8002354390000002 podStartE2EDuration="2.631989934s" totalImagesPullingTime="831.754495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:16.631447477 +0000 UTC m=+946.321330605" watchObservedRunningTime="2026-10-31 01:12:16.631989934 +0000 UTC m=+946.321873082"
Oct 31 01:12:17.610122 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:17.610092    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:18.500195 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:18.500151    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 01:12:19.970721 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:19.970689    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:22.523654 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:22.523623    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 01:12:24.786369 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:24.786324    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:24.966099 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:24.966063    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:25.152097 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:25.152020    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:25.981690 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:25.976629    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:29.180884 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:12:29.180851    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:12:29.797745 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:29.797717    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:34.503536 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:34.503483    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:36.154069 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:36.154039    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:39.007788 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:39.007738    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:39.486317 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:39.486280    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:39.693830 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:39.693803    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:39.885442 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:39.885400    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:41.146647 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:41.146614    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:43.429027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:43.428257    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:46712->10.133.0.58:15021: read: connection reset by peer" start-of-body=
Oct 31 01:12:43.429027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:43.428335    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:46712->10.133.0.58:15021: read: connection reset by peer"
Oct 31 01:12:43.644852 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:43.644786    4476 generic.go:356] "Generic (PLEG): container finished" podID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerID="57c12101cd32a0a9e60b93f3e966b3324e78680563b69db77d9ea95543da473f" exitCode=0
Oct 31 01:12:43.645055 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:43.644853    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" event={"ID":"7c71078d-3e0f-4e16-9df8-7cabafce5af9","Type":"ContainerDied","Data":"57c12101cd32a0a9e60b93f3e966b3324e78680563b69db77d9ea95543da473f"}
Oct 31 01:12:43.645288 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:43.645273    4476 scope.go:122] "RemoveContainer" containerID="57c12101cd32a0a9e60b93f3e966b3324e78680563b69db77d9ea95543da473f"
Oct 31 01:12:43.646590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:43.646564    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:12:43.646684 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:43.646639    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:12:43.646684 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:43.646670    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:12:44.385623 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:44.385592    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:44.651517 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:44.651441    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" event={"ID":"7c71078d-3e0f-4e16-9df8-7cabafce5af9","Type":"ContainerStarted","Data":"041e5e9ea3188212b09d71810fe253fa64f4182925545d8f4caf5bd609a847a9"}
Oct 31 01:12:45.651123 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:45.651087    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:12:45.651344 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:45.651322    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 01:12:45.651396 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:45.651369    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 01:12:46.167771 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.167731    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"]
Oct 31 01:12:46.177445 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.177416    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.211069 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.211044    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-home\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.211230 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.211097    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-model-cache\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.211285 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.211266    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2ck\" (UniqueName: \"kubernetes.io/projected/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-kube-api-access-xs2ck\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.211325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.211305    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-tmp-dir\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.211392 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.211376    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-tls-certs\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.211445 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.211433    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-dshm\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.228843 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.228811    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"]
Oct 31 01:12:46.282200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.280153    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-acd385\"/\"openshift-service-ca.crt\""
Oct 31 01:12:46.282200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.280307    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-acd385\"/\"kube-root-ca.crt\""
Oct 31 01:12:46.282200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.280472    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-acd385\"/\"default-dockercfg-pp82z\""
Oct 31 01:12:46.282200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.280540    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-acd385\"/\"premium-w4-acd385-kserve-self-signed-certs\""
Oct 31 01:12:46.312691 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.312659    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2ck\" (UniqueName: \"kubernetes.io/projected/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-kube-api-access-xs2ck\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.312691 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.312695    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-tmp-dir\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.312942 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.312723    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-tls-certs\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.312942 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.312765    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-dshm\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.312942 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.312844    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-home\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.313520 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.313271    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-tmp-dir\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.313520 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.313468    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-home\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.313822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.313801    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-model-cache\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.314132 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.314106    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-model-cache\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.316058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.316035    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-dshm\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.316587 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.316562    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-tls-certs\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.332249 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.332207    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:46.387692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.387647    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2ck\" (UniqueName: \"kubernetes.io/projected/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-kube-api-access-xs2ck\") pod \"premium-w4-acd385-kserve-65676959d8-579fq\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") " pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.499276 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.499163    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:46.651344 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.651314    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 01:12:46.651496 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.651358    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 01:12:46.881200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:46.877951    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"]
Oct 31 01:12:46.884694 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:12:46.884652    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b2d11a3_2014_4d3c_b1e6_c98233a16c74.slice/crio-7770ff8827139658923bde9db307e3f39e374418b038ea50e0dba14eec6ce497 WatchSource:0}: Error finding container 7770ff8827139658923bde9db307e3f39e374418b038ea50e0dba14eec6ce497: Status 404 returned error can't find the container with id 7770ff8827139658923bde9db307e3f39e374418b038ea50e0dba14eec6ce497
Oct 31 01:12:47.651654 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:47.651619    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 01:12:47.652054 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:47.651672    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 01:12:47.690712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:47.690675    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" event={"ID":"3b2d11a3-2014-4d3c-b1e6-c98233a16c74","Type":"ContainerStarted","Data":"e757081dce247a93711fa19a9d3fa20f1b8e72c6ab70d3fd5d2338e824e95083"}
Oct 31 01:12:47.690885 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:47.690726    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" event={"ID":"3b2d11a3-2014-4d3c-b1e6-c98233a16c74","Type":"ContainerStarted","Data":"7770ff8827139658923bde9db307e3f39e374418b038ea50e0dba14eec6ce497"}
Oct 31 01:12:47.809451 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:47.809388    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podStartSLOduration=1.809369837 podStartE2EDuration="1.809369837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:47.807403343 +0000 UTC m=+977.497286472" watchObservedRunningTime="2026-10-31 01:12:47.809369837 +0000 UTC m=+977.499252965"
Oct 31 01:12:48.499890 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:48.499860    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:48.651514 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:48.651481    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 01:12:48.651679 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:48.651541    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 01:12:49.194264 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:49.194195    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:49.652023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:49.651987    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 01:12:49.652447 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:49.652039    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 01:12:50.655630 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:50.655595    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:12:50.656125 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:50.655984    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:12:50.656720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:50.656698    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:12:52.954974 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:52.954942    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:53.545394 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:53.545366    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:53.788068 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:53.788032    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:54.144457 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:54.144427    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:56.500038 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:56.500012    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:56.512468 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:56.512447    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:56.758728 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:56.758636    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:12:58.790127 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:12:58.790090    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:59.179956 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:12:59.179880    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:13:03.491566 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:03.491530    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:08.178227 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:08.178192    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:09.986804 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:09.986770    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:13.018135 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:13.018098    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:16.112628 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:16.112592    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:17.714309 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:17.714279    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:19.169378 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.169345    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"]
Oct 31 01:13:19.173056 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.173040    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.276501 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.276470    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-97b317\"/\"kube-root-ca.crt\""
Oct 31 01:13:19.276703 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.276528    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-97b317\"/\"openshift-service-ca.crt\""
Oct 31 01:13:19.276703 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.276528    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-97b317\"/\"simulator-w1-97b317-kserve-self-signed-certs\""
Oct 31 01:13:19.276703 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.276560    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-97b317\"/\"default-dockercfg-64ksq\""
Oct 31 01:13:19.278660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.278632    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"]
Oct 31 01:13:19.298444 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.298416    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:19.337748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.337708    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wkxm\" (UniqueName: \"kubernetes.io/projected/cbdc50dd-29d0-4610-a361-cfd11ad1f501-kube-api-access-8wkxm\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.337748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.337748    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-dshm\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.338002 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.337901    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-tmp-dir\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.338002 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.337936    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-model-cache\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.338119 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.338055    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbdc50dd-29d0-4610-a361-cfd11ad1f501-tls-certs\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.338119 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.338082    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-home\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.438789 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.438702    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wkxm\" (UniqueName: \"kubernetes.io/projected/cbdc50dd-29d0-4610-a361-cfd11ad1f501-kube-api-access-8wkxm\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.438789 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.438746    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-dshm\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.439032 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.438847    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-tmp-dir\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.439032 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.438883    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-model-cache\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.439032 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.438961    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbdc50dd-29d0-4610-a361-cfd11ad1f501-tls-certs\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.439032 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.438987    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-home\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.439312 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.439287    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-tmp-dir\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.439604 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.439577    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-home\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.439730 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.439584    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-model-cache\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.441125 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.441105    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-dshm\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.441793 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.441769    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbdc50dd-29d0-4610-a361-cfd11ad1f501-tls-certs\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.494850 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.494814    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wkxm\" (UniqueName: \"kubernetes.io/projected/cbdc50dd-29d0-4610-a361-cfd11ad1f501-kube-api-access-8wkxm\") pod \"simulator-w1-97b317-kserve-74d844d897-vsrgk\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") " pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:19.785343 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:19.785307    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:20.159944 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:20.159919    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"]
Oct 31 01:13:20.161508 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:13:20.161478    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbdc50dd_29d0_4610_a361_cfd11ad1f501.slice/crio-ddffe24a5278a90dfa87103140aead9d0f24acd11c2a0c7cf1d009cc4a3fe5b6 WatchSource:0}: Error finding container ddffe24a5278a90dfa87103140aead9d0f24acd11c2a0c7cf1d009cc4a3fe5b6: Status 404 returned error can't find the container with id ddffe24a5278a90dfa87103140aead9d0f24acd11c2a0c7cf1d009cc4a3fe5b6
Oct 31 01:13:20.874347 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:20.874308    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" event={"ID":"cbdc50dd-29d0-4610-a361-cfd11ad1f501","Type":"ContainerStarted","Data":"ae972356d26187494eb58316cd76e3d9dafd9e1ac9fded1e3bc3940e89788eca"}
Oct 31 01:13:20.874739 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:20.874361    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" event={"ID":"cbdc50dd-29d0-4610-a361-cfd11ad1f501","Type":"ContainerStarted","Data":"ddffe24a5278a90dfa87103140aead9d0f24acd11c2a0c7cf1d009cc4a3fe5b6"}
Oct 31 01:13:20.989236 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:20.989166    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" podStartSLOduration=1.9891510079999999 podStartE2EDuration="1.989151008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:20.987567644 +0000 UTC m=+1010.677450775" watchObservedRunningTime="2026-10-31 01:13:20.989151008 +0000 UTC m=+1010.679034135"
Oct 31 01:13:21.785531 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:21.785502    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:22.586331 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:22.586298    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:27.371612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:27.371566    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:29.181023 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:13:29.180990    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:13:29.786189 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:29.786155    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:29.798712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:29.798688    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:29.936645 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:29.936617    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:13:30.899868 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:30.899828    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:31.640507 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:31.640471    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:31.866365 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:31.866332    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:32.521098 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:32.521072    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:36.503653 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:36.503615    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:37.161815 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:37.161783    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:41.882763 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:41.882685    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:42.668239 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:42.668156    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:46.593119 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:46.593064    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:51.188834 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:51.188780    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:51.365036 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:51.365005    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:55.985084 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:13:55.985050    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:59.180356 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:13:59.180319    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:14:00.776121 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:00.776091    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:05.470787 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:05.470739    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:10.182742 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:10.182700    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:14.885365 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:14.885330    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:15.090199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:15.087381    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:16.460410 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:16.460348    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:19.583026 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:19.582992    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:20.042034 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:20.042001    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:24.394765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:24.394731    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:26.280673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.280641    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"]
Oct 31 01:14:26.284709 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.284689    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.336319 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.336297    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-26fbc4-openshift-default-dockercfg-bscl8\""
Oct 31 01:14:26.379910 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.379879    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.380202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.380102    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.380392 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.380365    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.380504 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.380441    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.380577 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.380556    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.380626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.380609    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.380677 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.380644    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.380753 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.380732    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.380803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.380790    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np47s\" (UniqueName: \"kubernetes.io/projected/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-kube-api-access-np47s\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.388470 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.388442    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"]
Oct 31 01:14:26.481408 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.481380    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.481578 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.481420    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.481578 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.481471    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.481578 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.481508    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.481578 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.481532    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np47s\" (UniqueName: \"kubernetes.io/projected/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-kube-api-access-np47s\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.481730 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.481595    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.481730 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.481642    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.481730 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.481670    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.481874 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.481742    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.481934 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.481880    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.481995 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.481932    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.482156 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.482124    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.482262 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.482240    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.482330 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.482288    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.483948 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.483928    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.484048 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.484031    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.539065 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.538999    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.539226 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.539202    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np47s\" (UniqueName: \"kubernetes.io/projected/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-kube-api-access-np47s\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:26.597221 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:26.597170    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:27.020502 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:27.020471    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"]
Oct 31 01:14:27.022853 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:14:27.022819    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0eace8b_08cd_43a8_b4f9_c5eacce5135e.slice/crio-043cfbf93af79af8e419bebd3df8a28f17e10fe5971fc0062cb4d27819a3c35e WatchSource:0}: Error finding container 043cfbf93af79af8e419bebd3df8a28f17e10fe5971fc0062cb4d27819a3c35e: Status 404 returned error can't find the container with id 043cfbf93af79af8e419bebd3df8a28f17e10fe5971fc0062cb4d27819a3c35e
Oct 31 01:14:27.025653 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:27.025617    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:14:27.025767 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:27.025687    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:14:27.025767 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:27.025739    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:14:27.227246 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:27.227214    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph" event={"ID":"c0eace8b-08cd-43a8-b4f9-c5eacce5135e","Type":"ContainerStarted","Data":"673a311a3d58e21073793faba4b4d0c7b6ea28762506c1bbd520f4468b503b03"}
Oct 31 01:14:27.227407 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:27.227264    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph" event={"ID":"c0eace8b-08cd-43a8-b4f9-c5eacce5135e","Type":"ContainerStarted","Data":"043cfbf93af79af8e419bebd3df8a28f17e10fe5971fc0062cb4d27819a3c35e"}
Oct 31 01:14:27.345927 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:27.344628    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph" podStartSLOduration=1.344612188 podStartE2EDuration="1.344612188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:27.343716998 +0000 UTC m=+1077.033600162" watchObservedRunningTime="2026-10-31 01:14:27.344612188 +0000 UTC m=+1077.034495393"
Oct 31 01:14:27.602258 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:27.598284    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:28.599815 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:28.599777    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 01:14:28.600287 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:28.599836    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph" podUID="c0eace8b-08cd-43a8-b4f9-c5eacce5135e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 01:14:29.020993 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:29.020915    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:29.198843 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:29.198807    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:29.207563 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:14:29.207523    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:14:29.597869 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:29.597829    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 01:14:29.598022 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:29.597890    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph" podUID="c0eace8b-08cd-43a8-b4f9-c5eacce5135e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 01:14:30.597861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:30.597831    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 01:14:30.598279 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:30.597889    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph" podUID="c0eace8b-08cd-43a8-b4f9-c5eacce5135e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 01:14:31.598129 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:31.598097    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 01:14:31.598583 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:31.598160    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph" podUID="c0eace8b-08cd-43a8-b4f9-c5eacce5135e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 01:14:32.597729 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:32.597692    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 01:14:32.597899 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:32.597751    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph" podUID="c0eace8b-08cd-43a8-b4f9-c5eacce5135e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 01:14:33.797304 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:33.797014    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:33.797889 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:33.797865    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:33.799452 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:33.799420    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:14:33.842399 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:33.842168    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:36.820266 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:36.820235    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:38.588732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:38.588701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:43.293914 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:43.293882    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:48.016355 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:48.016305    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:52.774347 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:52.774304    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:57.615746 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:14:57.615701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:59.181082 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:14:59.181046    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:15:01.537256 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:01.537215    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:02.521391 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:02.521359    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:07.442466 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:07.442433    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:12.264431 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:12.264349    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:16.795065 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:16.795023    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:16.981168 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:16.981136    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:21.576533 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:21.576497    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:26.236538 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:26.236503    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:29.182209 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:15:29.182153    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:15:30.982984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:30.982939    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:35.780128 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:35.780086    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:37.160418 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:37.160382    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:37.599295 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:37.599264    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:37.778272 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:37.778241    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:40.598351 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:40.598317    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:45.310689 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:45.310649    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:46.309474 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:46.309447    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:46.487224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:46.487193    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:46.666555 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:46.666464    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:49.691805 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:49.691768    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:50.087091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.087061    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:50.174341 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.174308    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"]
Oct 31 01:15:50.177982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.177966    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:50.279892 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.279858    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 01:15:50.280043 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.279978    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 01:15:50.280043 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.280026    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 01:15:50.280158 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.280034    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-79935e0896e1\""
Oct 31 01:15:50.280158 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.280062    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-84267ab48818\""
Oct 31 01:15:50.280158 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.280093    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 01:15:50.280158 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.280107    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-xd8g9\""
Oct 31 01:15:50.282538 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.282513    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"]
Oct 31 01:15:50.317350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.317323    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefbd225-13da-4085-9088-865d4b331e3d-plugins-config-volume\") pod \"payload-processing-external-model-7fb874b485-9dxqg\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") " pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:50.317467 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.317357    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials\") pod \"payload-processing-external-model-7fb874b485-9dxqg\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") " pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:50.317467 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.317377    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cefbd225-13da-4085-9088-865d4b331e3d-routing-overlay-volume\") pod \"payload-processing-external-model-7fb874b485-9dxqg\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") " pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:50.317467 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.317401    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm5tx\" (UniqueName: \"kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-kube-api-access-gm5tx\") pod \"payload-processing-external-model-7fb874b485-9dxqg\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") " pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:50.418720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.418652    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefbd225-13da-4085-9088-865d4b331e3d-plugins-config-volume\") pod \"payload-processing-external-model-7fb874b485-9dxqg\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") " pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:50.418720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.418689    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials\") pod \"payload-processing-external-model-7fb874b485-9dxqg\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") " pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:50.418720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.418710    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cefbd225-13da-4085-9088-865d4b331e3d-routing-overlay-volume\") pod \"payload-processing-external-model-7fb874b485-9dxqg\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") " pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:50.418977 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.418739    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm5tx\" (UniqueName: \"kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-kube-api-access-gm5tx\") pod \"payload-processing-external-model-7fb874b485-9dxqg\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") " pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:50.419451 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.419430    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cefbd225-13da-4085-9088-865d4b331e3d-routing-overlay-volume\") pod \"payload-processing-external-model-7fb874b485-9dxqg\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") " pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:50.419600 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.419539    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefbd225-13da-4085-9088-865d4b331e3d-plugins-config-volume\") pod \"payload-processing-external-model-7fb874b485-9dxqg\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") " pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:50.421472 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.421454    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials\") pod \"payload-processing-external-model-7fb874b485-9dxqg\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") " pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:50.475267 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.475236    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm5tx\" (UniqueName: \"kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-kube-api-access-gm5tx\") pod \"payload-processing-external-model-7fb874b485-9dxqg\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") " pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:50.490846 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.490822    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:50.863157 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:50.863124    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"]
Oct 31 01:15:50.866764 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:15:50.866738    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefbd225_13da_4085_9088_865d4b331e3d.slice/crio-7951e1f98a20bb08ead9e59536dbb39a0528e19dce537d87e979f9d36da249c9 WatchSource:0}: Error finding container 7951e1f98a20bb08ead9e59536dbb39a0528e19dce537d87e979f9d36da249c9: Status 404 returned error can't find the container with id 7951e1f98a20bb08ead9e59536dbb39a0528e19dce537d87e979f9d36da249c9
Oct 31 01:15:51.687769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:51.687727    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg" event={"ID":"cefbd225-13da-4085-9088-865d4b331e3d","Type":"ContainerStarted","Data":"bbc965c9ccbdf14b9073a71b134df9e07267af63d4d7fd14d05fd6667eeb0b99"}
Oct 31 01:15:51.687971 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:51.687793    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg" event={"ID":"cefbd225-13da-4085-9088-865d4b331e3d","Type":"ContainerStarted","Data":"7951e1f98a20bb08ead9e59536dbb39a0528e19dce537d87e979f9d36da249c9"}
Oct 31 01:15:51.687971 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:51.687870    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:15:51.802163 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:51.802108    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg" podStartSLOduration=1.8020956030000002 podStartE2EDuration="1.802095603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:51.800212945 +0000 UTC m=+1161.490096074" watchObservedRunningTime="2026-10-31 01:15:51.802095603 +0000 UTC m=+1161.491978730"
Oct 31 01:15:54.764470 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:54.764439    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:59.181113 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:15:59.181076    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:15:59.472931 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:15:59.472844    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:01.876627 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:01.876592    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:02.695943 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:02.695914    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:16:03.179608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.179576    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"]
Oct 31 01:16:03.185706 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.185681    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.237052 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.236996    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"]
Oct 31 01:16:03.237854 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.237832    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ec695b\"/\"kube-root-ca.crt\""
Oct 31 01:16:03.238003 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.237985    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ec695b\"/\"openshift-service-ca.crt\""
Oct 31 01:16:03.238068 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.237995    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ec695b\"/\"default-dockercfg-87kps\""
Oct 31 01:16:03.238110 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.238084    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ec695b\"/\"e2e-conflict-a-c17e7941-kserve-self-signed-certs\""
Oct 31 01:16:03.244489 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.244453    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7b7q\" (UniqueName: \"kubernetes.io/projected/2ea8587f-d236-49e3-ae23-24ad6ad3a366-kube-api-access-j7b7q\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.244605 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.244511    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-dshm\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.244686 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.244665    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-tmp-dir\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.244792 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.244772    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-model-cache\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.244876 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.244859    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea8587f-d236-49e3-ae23-24ad6ad3a366-tls-certs\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.244916 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.244899    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-home\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.310446 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.310390    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:03.345890 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.345862    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea8587f-d236-49e3-ae23-24ad6ad3a366-tls-certs\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.346070 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.345902    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-home\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.346070 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.345936    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7b7q\" (UniqueName: \"kubernetes.io/projected/2ea8587f-d236-49e3-ae23-24ad6ad3a366-kube-api-access-j7b7q\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.346070 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.345970    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-dshm\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.346070 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.346062    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-tmp-dir\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.346317 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.346128    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-model-cache\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.346418 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.346398    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-home\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.346604 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.346574    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-model-cache\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.346734 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.346597    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-tmp-dir\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.348790 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.348765    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-dshm\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.348997 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.348978    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea8587f-d236-49e3-ae23-24ad6ad3a366-tls-certs\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.402497 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.402468    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7b7q\" (UniqueName: \"kubernetes.io/projected/2ea8587f-d236-49e3-ae23-24ad6ad3a366-kube-api-access-j7b7q\") pod \"e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.498049 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.497965    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:03.898000 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:03.897971    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"]
Oct 31 01:16:03.900138 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:16:03.900108    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ea8587f_d236_49e3_ae23_24ad6ad3a366.slice/crio-a6c32643039fe45a0f7d6d72555dc4416eaaa988ac3194adb2c93d550e6d7120 WatchSource:0}: Error finding container a6c32643039fe45a0f7d6d72555dc4416eaaa988ac3194adb2c93d550e6d7120: Status 404 returned error can't find the container with id a6c32643039fe45a0f7d6d72555dc4416eaaa988ac3194adb2c93d550e6d7120
Oct 31 01:16:04.304783 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:04.304749    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:04.776006 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:04.775917    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt" event={"ID":"2ea8587f-d236-49e3-ae23-24ad6ad3a366","Type":"ContainerStarted","Data":"caba281289f9fbfa03d7faf32de3ecf0b4685820de2304e68d630a67d574b4a6"}
Oct 31 01:16:04.776006 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:04.775967    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt" event={"ID":"2ea8587f-d236-49e3-ae23-24ad6ad3a366","Type":"ContainerStarted","Data":"a6c32643039fe45a0f7d6d72555dc4416eaaa988ac3194adb2c93d550e6d7120"}
Oct 31 01:16:04.890324 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:04.890258    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt" podStartSLOduration=1.890242932 podStartE2EDuration="1.890242932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:04.889157051 +0000 UTC m=+1174.579040179" watchObservedRunningTime="2026-10-31 01:16:04.890242932 +0000 UTC m=+1174.580126058"
Oct 31 01:16:05.498350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:05.498313    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:08.989224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:08.989167    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:13.498574 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:13.498538    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:13.511170 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:13.511143    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:13.677613 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:13.677586    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:13.838042 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:13.838007    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:16:17.131594 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.131563    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:17.318767 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.318737    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"]
Oct 31 01:16:17.322406 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.322389    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.388843 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.388780    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.388843 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.388815    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.388843 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.388833    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.389065 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.388912    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.389065 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.388989    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.389065 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.389011    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.389206 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.389124    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.389206 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.389166    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwbvm\" (UniqueName: \"kubernetes.io/projected/7dc706e1-fe7c-4e88-88a2-ef35706e2791-kube-api-access-qwbvm\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.389329 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.389308    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.426784 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.426760    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"]
Oct 31 01:16:17.490454 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.490432    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.490628 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.490464    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.490628 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.490482    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.490628 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.490504    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.490628 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.490528    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.490855 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.490653    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.490855 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.490773    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.490855 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.490830    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwbvm\" (UniqueName: \"kubernetes.io/projected/7dc706e1-fe7c-4e88-88a2-ef35706e2791-kube-api-access-qwbvm\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.490998 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.490884    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.490998 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.490901    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.490998 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.490920    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.491115 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.491012    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.491152 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.491126    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.491455 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.491436    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.493216 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.493168    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.493400 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.493384    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.549408 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.549385    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.549549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.549522    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwbvm\" (UniqueName: \"kubernetes.io/projected/7dc706e1-fe7c-4e88-88a2-ef35706e2791-kube-api-access-qwbvm\") pod \"maas-default-gateway-openshift-default-fc8c56547-sxrp7\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:17.636374 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:17.636349    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:18.062020 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:18.061996    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"]
Oct 31 01:16:18.063363 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:16:18.063336    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc706e1_fe7c_4e88_88a2_ef35706e2791.slice/crio-3c3a7501317f5e6e5831190d48a1aefa322d24148b5c86d92ea09e740213e0b9 WatchSource:0}: Error finding container 3c3a7501317f5e6e5831190d48a1aefa322d24148b5c86d92ea09e740213e0b9: Status 404 returned error can't find the container with id 3c3a7501317f5e6e5831190d48a1aefa322d24148b5c86d92ea09e740213e0b9
Oct 31 01:16:18.065698 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:18.065662    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:16:18.065763 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:18.065733    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:16:18.065799 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:18.065769    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:16:18.166795 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:18.166770    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:18.388852 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:18.388774    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:18.890417 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:18.890385    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7" event={"ID":"7dc706e1-fe7c-4e88-88a2-ef35706e2791","Type":"ContainerStarted","Data":"3a5ba4a034839af488cd0969a6f3a5ed18e7ec0f65726c6ee9b8546508447856"}
Oct 31 01:16:18.890608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:18.890439    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7" event={"ID":"7dc706e1-fe7c-4e88-88a2-ef35706e2791","Type":"ContainerStarted","Data":"3c3a7501317f5e6e5831190d48a1aefa322d24148b5c86d92ea09e740213e0b9"}
Oct 31 01:16:19.056622 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:19.056566    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7" podStartSLOduration=2.056551993 podStartE2EDuration="2.056551993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:19.055135324 +0000 UTC m=+1188.745018452" watchObservedRunningTime="2026-10-31 01:16:19.056551993 +0000 UTC m=+1188.746435139"
Oct 31 01:16:19.636930 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:19.636894    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:19.638473 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:19.638447    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 01:16:19.638587 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:19.638494    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7" podUID="7dc706e1-fe7c-4e88-88a2-ef35706e2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 01:16:20.636897 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:20.636857    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 01:16:20.637076 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:20.636941    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7" podUID="7dc706e1-fe7c-4e88-88a2-ef35706e2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 01:16:21.636938 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:21.636904    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 01:16:21.637120 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:21.636971    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7" podUID="7dc706e1-fe7c-4e88-88a2-ef35706e2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 01:16:22.637204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:22.637147    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 01:16:22.637614 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:22.637213    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7" podUID="7dc706e1-fe7c-4e88-88a2-ef35706e2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 01:16:23.088187 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:23.088134    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:23.637569 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:23.637535    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 01:16:23.637966 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:23.637589    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7" podUID="7dc706e1-fe7c-4e88-88a2-ef35706e2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 01:16:24.637015 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:24.636980    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 01:16:24.637237 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:24.637037    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7" podUID="7dc706e1-fe7c-4e88-88a2-ef35706e2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 01:16:25.636798 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:25.636761    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxrp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 01:16:25.637338 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:25.636822    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7" podUID="7dc706e1-fe7c-4e88-88a2-ef35706e2791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 01:16:26.660259 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:26.660224    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:26.660784 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:26.660656    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:26.661482 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:26.661463    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:16:27.758556 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:27.758526    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:29.180503 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:16:29.180462    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:16:30.560744 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:30.560708    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:16:30.575196 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:30.575091    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:16:32.572981 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.572939    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:32.694077 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.694041    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"]
Oct 31 01:16:32.699485 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.699464    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.753661 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.753633    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"]
Oct 31 01:16:32.788207 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.788152    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-home\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.788346 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.788286    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xkpm\" (UniqueName: \"kubernetes.io/projected/27377642-d40d-4498-af5d-16ca6a58ef54-kube-api-access-2xkpm\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.788394 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.788360    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-model-cache\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.788394 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.788384    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27377642-d40d-4498-af5d-16ca6a58ef54-tls-certs\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.788489 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.788407    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-dshm\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.788539 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.788524    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-tmp-dir\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.800059 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.800043    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-97b317\"/\"distinct-w1-97b317-kserve-self-signed-certs\""
Oct 31 01:16:32.822268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.822238    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:32.889697 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.889607    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-tmp-dir\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.889697 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.889659    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-home\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.889930 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.889718    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xkpm\" (UniqueName: \"kubernetes.io/projected/27377642-d40d-4498-af5d-16ca6a58ef54-kube-api-access-2xkpm\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.889930 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.889869    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-model-cache\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.889930 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.889924    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27377642-d40d-4498-af5d-16ca6a58ef54-tls-certs\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.890083 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.889975    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-dshm\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.890083 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.890059    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-tmp-dir\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.890150 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.890080    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-home\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.890300 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.890281    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-model-cache\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.892443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.892411    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-dshm\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.892666 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.892651    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27377642-d40d-4498-af5d-16ca6a58ef54-tls-certs\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:32.946548 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:32.946518    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xkpm\" (UniqueName: \"kubernetes.io/projected/27377642-d40d-4498-af5d-16ca6a58ef54-kube-api-access-2xkpm\") pod \"distinct-w1-97b317-kserve-5566f4dc75-str28\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:33.010422 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:33.010399    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:33.385485 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:33.385464    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"]
Oct 31 01:16:33.387200 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:16:33.387157    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27377642_d40d_4498_af5d_16ca6a58ef54.slice/crio-15f7546643e606384cfe38d10cf63e72bcbe062dfb5ac8a5d48173f1f3173dca WatchSource:0}: Error finding container 15f7546643e606384cfe38d10cf63e72bcbe062dfb5ac8a5d48173f1f3173dca: Status 404 returned error can't find the container with id 15f7546643e606384cfe38d10cf63e72bcbe062dfb5ac8a5d48173f1f3173dca
Oct 31 01:16:33.987518 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:33.987476    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" event={"ID":"27377642-d40d-4498-af5d-16ca6a58ef54","Type":"ContainerStarted","Data":"51676a73d4dd485f021f44784500662f8c1074b6baefd3724687ea1d6b73b65e"}
Oct 31 01:16:33.987518 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:33.987524    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" event={"ID":"27377642-d40d-4498-af5d-16ca6a58ef54","Type":"ContainerStarted","Data":"15f7546643e606384cfe38d10cf63e72bcbe062dfb5ac8a5d48173f1f3173dca"}
Oct 31 01:16:34.011562 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:34.011526    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:34.102342 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:34.102294    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" podStartSLOduration=2.102280755 podStartE2EDuration="2.102280755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:34.100104777 +0000 UTC m=+1203.789987904" watchObservedRunningTime="2026-10-31 01:16:34.102280755 +0000 UTC m=+1203.792163878"
Oct 31 01:16:37.366164 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:37.366075    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:37.550335 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:37.550302    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:37.731711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:37.731635    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:42.280583 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.280500    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:42.633272 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.633193    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"]
Oct 31 01:16:42.637291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.637261    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.686381 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.686343    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.686558 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.686487    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.686618 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.686566    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.686618 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.686598    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.686763 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.686745    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.686808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.686794    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb63a95e-790b-425c-bb97-9925b8f20a49-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.686886 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.686864    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twpzn\" (UniqueName: \"kubernetes.io/projected/fb63a95e-790b-425c-bb97-9925b8f20a49-kube-api-access-twpzn\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.686962 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.686944    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.687039 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.686978    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.697144 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.697118    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"]
Oct 31 01:16:42.788619 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.788574    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.788849 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.788635    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.789155 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.789135    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.789403 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.789386    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.789545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.789531    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb63a95e-790b-425c-bb97-9925b8f20a49-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.789678 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.789661    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twpzn\" (UniqueName: \"kubernetes.io/projected/fb63a95e-790b-425c-bb97-9925b8f20a49-kube-api-access-twpzn\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.789803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.789788    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.789905 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.789890    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.790095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.790077    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.790575 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.790541    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.790663 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.790621    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.790765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.790744    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb63a95e-790b-425c-bb97-9925b8f20a49-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.790943 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.790916    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.791010 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.790926    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.792312 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.792289    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.792528 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.792512    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.847133 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.847108    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twpzn\" (UniqueName: \"kubernetes.io/projected/fb63a95e-790b-425c-bb97-9925b8f20a49-kube-api-access-twpzn\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.847266 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.847110    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:42.950920 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:42.950855    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:43.011550 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:43.011515    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:43.027974 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:43.027952    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:43.048460 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:43.048434    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:16:43.419159 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:43.419137    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"]
Oct 31 01:16:43.420822 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:16:43.420798    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb63a95e_790b_425c_bb97_9925b8f20a49.slice/crio-585bbcfc441c5fb07d8adaabe5a5361d83a5bafd3b91cc2ad814973b66392daf WatchSource:0}: Error finding container 585bbcfc441c5fb07d8adaabe5a5361d83a5bafd3b91cc2ad814973b66392daf: Status 404 returned error can't find the container with id 585bbcfc441c5fb07d8adaabe5a5361d83a5bafd3b91cc2ad814973b66392daf
Oct 31 01:16:43.423051 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:43.423020    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:16:43.423122 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:43.423099    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:16:43.423159 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:43.423129    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:16:44.043534 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:44.043499    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7" event={"ID":"fb63a95e-790b-425c-bb97-9925b8f20a49","Type":"ContainerStarted","Data":"53bfa36de2857038f68263e6b016e60a9e1f3342aaf1fd7038c0f932d31feec0"}
Oct 31 01:16:44.043747 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:44.043560    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7" event={"ID":"fb63a95e-790b-425c-bb97-9925b8f20a49","Type":"ContainerStarted","Data":"585bbcfc441c5fb07d8adaabe5a5361d83a5bafd3b91cc2ad814973b66392daf"}
Oct 31 01:16:44.162780 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:44.162726    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7" podStartSLOduration=2.16271302 podStartE2EDuration="2.16271302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:44.160436802 +0000 UTC m=+1213.850319931" watchObservedRunningTime="2026-10-31 01:16:44.16271302 +0000 UTC m=+1213.852596146"
Oct 31 01:16:44.952005 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:44.951973    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:44.953351 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:44.953326    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:16:44.953477 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:44.953372    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7" podUID="fb63a95e-790b-425c-bb97-9925b8f20a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 01:16:45.951519 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:45.951489    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:16:45.951696 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:45.951538    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7" podUID="fb63a95e-790b-425c-bb97-9925b8f20a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 01:16:46.951541 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:46.951507    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:16:46.951963 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:46.951560    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7" podUID="fb63a95e-790b-425c-bb97-9925b8f20a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 01:16:47.087202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.086812    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:47.558290 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.558254    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"]
Oct 31 01:16:47.584450 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.584418    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.642750 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.642721    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rtf\" (UniqueName: \"kubernetes.io/projected/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-kube-api-access-g6rtf\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.642917 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.642759    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.642917 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.642779    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.642917 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.642901    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.643062 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.642977    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.643062 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.643025    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.643152 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.643079    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.643152 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.643096    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.643302 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.643210    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.674018 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.673982    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"]
Oct 31 01:16:47.744358 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.744329    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.744358 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.744362    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.744586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.744398    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.744586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.744496    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rtf\" (UniqueName: \"kubernetes.io/projected/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-kube-api-access-g6rtf\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.744586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.744532    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.744586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.744557    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.744789 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.744608    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.744789 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.744694    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.744789 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.744732    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.744789 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.744764    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.745460 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.744954    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.745460 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.745075    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.745460 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.745220    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.745460 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.745406    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.746943 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.746918    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.747072 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.747054    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.800972 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.800938    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.801103 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.800939    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rtf\" (UniqueName: \"kubernetes.io/projected/b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8-kube-api-access-g6rtf\") pod \"maas-default-gateway-openshift-default-fc8c56547-gbwwl\" (UID: \"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.898039 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.897960    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:47.953644 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.953615    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:16:47.954318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:47.953684    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7" podUID="fb63a95e-790b-425c-bb97-9925b8f20a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 01:16:48.325951 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:48.325923    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"]
Oct 31 01:16:48.326736 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:16:48.326712    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85531e9_78a0_4f6a_bbbd_6622c2d5c7a8.slice/crio-922e388d1b3a0fb8161a770ba122e8e41c96b56922f5823251498503aeb9ad04 WatchSource:0}: Error finding container 922e388d1b3a0fb8161a770ba122e8e41c96b56922f5823251498503aeb9ad04: Status 404 returned error can't find the container with id 922e388d1b3a0fb8161a770ba122e8e41c96b56922f5823251498503aeb9ad04
Oct 31 01:16:48.329149 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:48.329118    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:16:48.329252 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:48.329212    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:16:48.329293 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:48.329268    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:16:48.951882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:48.951846    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:16:48.952065 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:48.951906    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7" podUID="fb63a95e-790b-425c-bb97-9925b8f20a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 01:16:49.116948 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:49.116912    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl" event={"ID":"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8","Type":"ContainerStarted","Data":"a2299e867b2dd00ee10c22b56977928a671b261428f5e66ab58f14d391358e9b"}
Oct 31 01:16:49.117543 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:49.116986    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl" event={"ID":"b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8","Type":"ContainerStarted","Data":"922e388d1b3a0fb8161a770ba122e8e41c96b56922f5823251498503aeb9ad04"}
Oct 31 01:16:49.238160 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:49.238053    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl" podStartSLOduration=2.238039497 podStartE2EDuration="2.238039497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:49.235764843 +0000 UTC m=+1218.925647970" watchObservedRunningTime="2026-10-31 01:16:49.238039497 +0000 UTC m=+1218.927922696"
Oct 31 01:16:49.899148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:49.899115    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:49.900838 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:49.900815    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-gbwwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:16:49.900966 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:49.900860    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl" podUID="b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 01:16:49.951783 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:49.951751    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:16:49.951919 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:49.951801    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7" podUID="fb63a95e-790b-425c-bb97-9925b8f20a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 01:16:50.339956 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:50.339925    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:50.898690 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:50.898656    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-gbwwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:16:50.898865 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:50.898709    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl" podUID="b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 01:16:50.909618 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:50.909587    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:50.955863 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:50.955843    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:50.956138 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:50.956118    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:50.956941 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:50.956925    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:16:51.087432 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:51.087391    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:51.775418 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:51.775383    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:51.898469 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:51.898432    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-gbwwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:16:51.898646 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:51.898492    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl" podUID="b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 01:16:52.898861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:52.898826    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-gbwwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:16:52.899261 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:52.898881    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl" podUID="b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 01:16:53.899366 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:53.899329    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-gbwwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:16:53.899806 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:53.899396    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl" podUID="b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 01:16:54.898491 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:54.898460    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-gbwwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:16:54.898704 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:54.898518    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl" podUID="b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 01:16:55.898755 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:55.898717    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-gbwwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:16:55.899158 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:55.898803    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl" podUID="b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 01:16:56.534538 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:56.534508    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:56.902567 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:56.902495    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:56.902934 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:56.902764    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:56.903486 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:16:56.903470    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-gbwwl"
Oct 31 01:16:59.182089 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:16:59.182056    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:17:01.322281 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:01.322246    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:02.209755 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:02.209720    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:02.732225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:02.732192    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:05.990591 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:05.990548    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:07.526165 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:07.526133    4476 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-79935e0896e1: secret "agc-cred-79935e0896e1" not found
Oct 31 01:17:07.526165 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:07.526166    4476 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg: secret "agc-cred-79935e0896e1" not found
Oct 31 01:17:07.526740 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:07.526260    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials podName:cefbd225-13da-4085-9088-865d4b331e3d nodeName:}" failed. No retries permitted until 2026-10-31 01:17:08.026239225 +0000 UTC m=+1237.716122330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials") pod "payload-processing-external-model-7fb874b485-9dxqg" (UID: "cefbd225-13da-4085-9088-865d4b331e3d") : secret "agc-cred-79935e0896e1" not found
Oct 31 01:17:08.032974 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:08.032943    4476 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-79935e0896e1: secret "agc-cred-79935e0896e1" not found
Oct 31 01:17:08.032974 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:08.032971    4476 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg: secret "agc-cred-79935e0896e1" not found
Oct 31 01:17:08.033153 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:08.033024    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials podName:cefbd225-13da-4085-9088-865d4b331e3d nodeName:}" failed. No retries permitted until 2026-10-31 01:17:09.033008177 +0000 UTC m=+1238.722891283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials") pod "payload-processing-external-model-7fb874b485-9dxqg" (UID: "cefbd225-13da-4085-9088-865d4b331e3d") : secret "agc-cred-79935e0896e1" not found
Oct 31 01:17:09.045351 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:09.045323    4476 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-79935e0896e1: secret "agc-cred-79935e0896e1" not found
Oct 31 01:17:09.045351 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:09.045350    4476 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg: secret "agc-cred-79935e0896e1" not found
Oct 31 01:17:09.045768 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:09.045396    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials podName:cefbd225-13da-4085-9088-865d4b331e3d nodeName:}" failed. No retries permitted until 2026-10-31 01:17:11.045382014 +0000 UTC m=+1240.735265123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials") pod "payload-processing-external-model-7fb874b485-9dxqg" (UID: "cefbd225-13da-4085-9088-865d4b331e3d") : secret "agc-cred-79935e0896e1" not found
Oct 31 01:17:10.667969 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:10.667939    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:11.068757 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:11.068730    4476 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-84267ab48818: secret "agc-cred-84267ab48818" not found
Oct 31 01:17:11.068757 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:11.068752    4476 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg: secret "agc-cred-84267ab48818" not found
Oct 31 01:17:11.068968 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:11.068813    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials podName:cefbd225-13da-4085-9088-865d4b331e3d nodeName:}" failed. No retries permitted until 2026-10-31 01:17:15.068798921 +0000 UTC m=+1244.758682030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials") pod "payload-processing-external-model-7fb874b485-9dxqg" (UID: "cefbd225-13da-4085-9088-865d4b331e3d") : secret "agc-cred-84267ab48818" not found
Oct 31 01:17:15.114883 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:15.114849    4476 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-84267ab48818: secret "agc-cred-84267ab48818" not found
Oct 31 01:17:15.114883 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:15.114875    4476 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg: secret "agc-cred-84267ab48818" not found
Oct 31 01:17:15.115477 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:15.114937    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials podName:cefbd225-13da-4085-9088-865d4b331e3d nodeName:}" failed. No retries permitted until 2026-10-31 01:17:23.114916562 +0000 UTC m=+1252.804799670 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials") pod "payload-processing-external-model-7fb874b485-9dxqg" (UID: "cefbd225-13da-4085-9088-865d4b331e3d") : secret "agc-cred-84267ab48818" not found
Oct 31 01:17:15.302345 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:15.302306    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:17.486999 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:17.486963    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:20.054111 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:20.054073    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:22.640244 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.640213    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"]
Oct 31 01:17:22.643950 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.643935    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.746227 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.746164    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ec695b\"/\"e2e-conflict-b-c17e7941-kserve-self-signed-certs\""
Oct 31 01:17:22.748689 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.748665    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"]
Oct 31 01:17:22.766940 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.766912    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:22.803413 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.803365    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-tmp-dir\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.803591 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.803579    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219c8fed-42af-45a6-a2d9-e95a794995a4-tls-certs\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.803646 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.803619    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-model-cache\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.803782 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.803763    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7m6l\" (UniqueName: \"kubernetes.io/projected/219c8fed-42af-45a6-a2d9-e95a794995a4-kube-api-access-h7m6l\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.803880 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.803860    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-dshm\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.803954 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.803936    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-home\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.904741 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.904664    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219c8fed-42af-45a6-a2d9-e95a794995a4-tls-certs\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.904741 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.904704    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-model-cache\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.904952 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.904761    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7m6l\" (UniqueName: \"kubernetes.io/projected/219c8fed-42af-45a6-a2d9-e95a794995a4-kube-api-access-h7m6l\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.904952 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.904792    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-dshm\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.904952 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.904815    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-home\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.904952 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.904937    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-tmp-dir\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.905293 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.905266    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-model-cache\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.905386 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.905312    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-home\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.905386 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.905346    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-tmp-dir\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.907157 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.907134    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-dshm\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.907448 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.907432    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219c8fed-42af-45a6-a2d9-e95a794995a4-tls-certs\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:22.960737 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:22.960715    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7m6l\" (UniqueName: \"kubernetes.io/projected/219c8fed-42af-45a6-a2d9-e95a794995a4-kube-api-access-h7m6l\") pod \"e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") " pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:23.208212 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:23.208106    4476 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-84267ab48818: secret "agc-cred-84267ab48818" not found
Oct 31 01:17:23.208212 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:23.208136    4476 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg: secret "agc-cred-84267ab48818" not found
Oct 31 01:17:23.208454 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:23.208228    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials podName:cefbd225-13da-4085-9088-865d4b331e3d nodeName:}" failed. No retries permitted until 2026-10-31 01:17:39.208207752 +0000 UTC m=+1268.898090862 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials") pod "payload-processing-external-model-7fb874b485-9dxqg" (UID: "cefbd225-13da-4085-9088-865d4b331e3d") : secret "agc-cred-84267ab48818" not found
Oct 31 01:17:23.256301 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:23.256273    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:23.631710 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:23.631674    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"]
Oct 31 01:17:23.632287 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:17:23.632254    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod219c8fed_42af_45a6_a2d9_e95a794995a4.slice/crio-09f992b262e234bcfd4bfca966d8c5f45fb294919b9e90d7b14913ca24d08189 WatchSource:0}: Error finding container 09f992b262e234bcfd4bfca966d8c5f45fb294919b9e90d7b14913ca24d08189: Status 404 returned error can't find the container with id 09f992b262e234bcfd4bfca966d8c5f45fb294919b9e90d7b14913ca24d08189
Oct 31 01:17:24.327450 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:24.327407    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c" event={"ID":"219c8fed-42af-45a6-a2d9-e95a794995a4","Type":"ContainerStarted","Data":"d8d03ac8dcb292434966074ed180c3a61ccea80e1454a4ee17a1bf97d9f15310"}
Oct 31 01:17:24.327941 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:24.327462    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c" event={"ID":"219c8fed-42af-45a6-a2d9-e95a794995a4","Type":"ContainerStarted","Data":"09f992b262e234bcfd4bfca966d8c5f45fb294919b9e90d7b14913ca24d08189"}
Oct 31 01:17:24.442465 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:24.442414    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c" podStartSLOduration=2.442399227 podStartE2EDuration="2.442399227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:24.440395828 +0000 UTC m=+1254.130278955" watchObservedRunningTime="2026-10-31 01:17:24.442399227 +0000 UTC m=+1254.132282354"
Oct 31 01:17:24.803220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:24.801688    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:25.257287 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:25.257210    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:29.181725 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:29.181689    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:17:29.610080 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:29.610046    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:33.256486 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:33.256450    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:33.268890 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:33.268864    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:33.403954 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:33.403918    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:17:34.283729 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:34.283694    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:38.983063 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:38.983017    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:39.308611 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:39.308582    4476 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-84267ab48818: secret "agc-cred-84267ab48818" not found
Oct 31 01:17:39.308611 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:39.308606    4476 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg: secret "agc-cred-84267ab48818" not found
Oct 31 01:17:39.308805 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:39.308663    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials podName:cefbd225-13da-4085-9088-865d4b331e3d nodeName:}" failed. No retries permitted until 2026-10-31 01:18:11.30865061 +0000 UTC m=+1300.998533715 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials") pod "payload-processing-external-model-7fb874b485-9dxqg" (UID: "cefbd225-13da-4085-9088-865d4b331e3d") : secret "agc-cred-84267ab48818" not found
Oct 31 01:17:43.678861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:43.678827    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:48.490265 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:48.490229    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:51.473909 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:51.473879    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:53.174075 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:53.174042    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:57.972276 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:57.972240    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:58.195923 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.195891    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"]
Oct 31 01:17:58.199764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.199741    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.287686 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.287650    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"]
Oct 31 01:17:58.310329 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.310298    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkg9g\" (UniqueName: \"kubernetes.io/projected/689153b0-2780-48a8-b936-7b96ba9ca8b1-kube-api-access-nkg9g\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.310488 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.310426    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.310488 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.310473    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.310582 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.310514    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.310682 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.310657    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.310735 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.310717    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.310816 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.310797    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.310853 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.310831    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689153b0-2780-48a8-b936-7b96ba9ca8b1-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.310897 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.310862    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.412156 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.412124    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.412156 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.412160    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.412402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.412292    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.412402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.412350    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.412402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.412393    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.412573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.412464    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.412573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.412478    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.412573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.412500    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689153b0-2780-48a8-b936-7b96ba9ca8b1-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.412573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.412527    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.412573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.412539    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.412828 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.412716    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.412828 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.412723    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkg9g\" (UniqueName: \"kubernetes.io/projected/689153b0-2780-48a8-b936-7b96ba9ca8b1-kube-api-access-nkg9g\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.412936 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.412882    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.413155 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.413078    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689153b0-2780-48a8-b936-7b96ba9ca8b1-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.414804 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.414777    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.414926 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.414866    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.468869 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.468836    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkg9g\" (UniqueName: \"kubernetes.io/projected/689153b0-2780-48a8-b936-7b96ba9ca8b1-kube-api-access-nkg9g\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.469016 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.468844    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.512858 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.512830    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:17:58.937863 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.937833    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"]
Oct 31 01:17:58.939211 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:17:58.939156    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689153b0_2780_48a8_b936_7b96ba9ca8b1.slice/crio-d8fa49a4f08fb15a35b744aae40e5d9e243fb93565b980b5b2332daf1cbccaea WatchSource:0}: Error finding container d8fa49a4f08fb15a35b744aae40e5d9e243fb93565b980b5b2332daf1cbccaea: Status 404 returned error can't find the container with id d8fa49a4f08fb15a35b744aae40e5d9e243fb93565b980b5b2332daf1cbccaea
Oct 31 01:17:58.941450 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.941414    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:17:58.941532 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.941483    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:17:58.941532 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:58.941520    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:17:59.181823 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:17:59.181785    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:17:59.532409 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:59.532370    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6" event={"ID":"689153b0-2780-48a8-b936-7b96ba9ca8b1","Type":"ContainerStarted","Data":"834a22d5c34b838ae06cd9e04954832bc73bb53e54e8779d13809b00229a07b8"}
Oct 31 01:17:59.532594 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:59.532432    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6" event={"ID":"689153b0-2780-48a8-b936-7b96ba9ca8b1","Type":"ContainerStarted","Data":"d8fa49a4f08fb15a35b744aae40e5d9e243fb93565b980b5b2332daf1cbccaea"}
Oct 31 01:17:59.649381 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:17:59.649331    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6" podStartSLOduration=1.649316752 podStartE2EDuration="1.649316752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:59.646551205 +0000 UTC m=+1289.336434332" watchObservedRunningTime="2026-10-31 01:17:59.649316752 +0000 UTC m=+1289.339199876"
Oct 31 01:18:00.513422 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:00.513389    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:18:00.515262 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:00.515238    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:18:00.515358 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:00.515290    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6" podUID="689153b0-2780-48a8-b936-7b96ba9ca8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 01:18:01.514279 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:01.514246    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:18:01.514639 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:01.514300    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6" podUID="689153b0-2780-48a8-b936-7b96ba9ca8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 01:18:02.513710 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:02.513678    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:18:02.513891 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:02.513738    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6" podUID="689153b0-2780-48a8-b936-7b96ba9ca8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 01:18:02.577956 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:02.577920    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:02.782123 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:02.782060    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:03.513905 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:03.513875    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:18:03.514074 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:03.513929    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6" podUID="689153b0-2780-48a8-b936-7b96ba9ca8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 01:18:04.514005 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:04.513972    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:18:04.514438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:04.514022    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6" podUID="689153b0-2780-48a8-b936-7b96ba9ca8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 01:18:05.518195 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:05.518140    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:18:05.518701 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:05.518232    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6" podUID="689153b0-2780-48a8-b936-7b96ba9ca8b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 01:18:06.523202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:06.522856    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:18:06.525069 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:06.524391    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:18:06.525289 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:06.525265    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:18:07.488087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:07.488054    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:07.918362 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:07.918324    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:09.537461 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.537424    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"]
Oct 31 01:18:09.541361 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.541345    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.567611 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.567573    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-model-cache\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.567770 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.567644    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv5xl\" (UniqueName: \"kubernetes.io/projected/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-kube-api-access-xv5xl\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.567770 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.567711    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-dshm\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.567770 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.567766    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-home\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.567927 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.567911    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-tmp-dir\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.567984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.567952    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-tls-certs\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.644041 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.644009    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-97b317\"/\"distinct-2-w1-97b317-kserve-self-signed-certs\""
Oct 31 01:18:09.646581 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.646561    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"]
Oct 31 01:18:09.666197 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.666148    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:09.669701 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.669668    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-model-cache\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.669818 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.669735    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv5xl\" (UniqueName: \"kubernetes.io/projected/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-kube-api-access-xv5xl\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.669818 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.669765    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-dshm\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.669818 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.669803    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-home\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.669988 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.669902    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-tmp-dir\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.669988 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.669938    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-tls-certs\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.670116 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.670094    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-model-cache\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.670310 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.670287    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-home\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.670383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.670319    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-tmp-dir\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.672352 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.672326    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-dshm\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.672604 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.672579    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-tls-certs\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.726199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.726149    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv5xl\" (UniqueName: \"kubernetes.io/projected/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-kube-api-access-xv5xl\") pod \"distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") " pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:09.854134 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:09.854051    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:10.227398 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:10.227378    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"]
Oct 31 01:18:10.229044 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:18:10.229011    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod762230e3_0692_4a5b_8e29_13bf8a3a4b3c.slice/crio-9fec7f80e694370f7989d5f4e914527ee02da719e4f4fea7b6fcdfe4e9f54018 WatchSource:0}: Error finding container 9fec7f80e694370f7989d5f4e914527ee02da719e4f4fea7b6fcdfe4e9f54018: Status 404 returned error can't find the container with id 9fec7f80e694370f7989d5f4e914527ee02da719e4f4fea7b6fcdfe4e9f54018
Oct 31 01:18:10.603356 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:10.603317    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4" event={"ID":"762230e3-0692-4a5b-8e29-13bf8a3a4b3c","Type":"ContainerStarted","Data":"e404de57619084613d3f871e3b5a8ecabe5bc8d81c7a3864f18a5a3726bfd27b"}
Oct 31 01:18:10.603819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:10.603368    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4" event={"ID":"762230e3-0692-4a5b-8e29-13bf8a3a4b3c","Type":"ContainerStarted","Data":"9fec7f80e694370f7989d5f4e914527ee02da719e4f4fea7b6fcdfe4e9f54018"}
Oct 31 01:18:10.854369 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:10.854251    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:11.391785 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:18:11.391758    4476 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-84267ab48818: secret "agc-cred-84267ab48818" not found
Oct 31 01:18:11.391785 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:18:11.391782    4476 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg: secret "agc-cred-84267ab48818" not found
Oct 31 01:18:11.392001 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:18:11.391845    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials podName:cefbd225-13da-4085-9088-865d4b331e3d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:15.391825567 +0000 UTC m=+1365.081708675 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials") pod "payload-processing-external-model-7fb874b485-9dxqg" (UID: "cefbd225-13da-4085-9088-865d4b331e3d") : secret "agc-cred-84267ab48818" not found
Oct 31 01:18:12.137086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:12.137043    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:16.783409 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:16.783375    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:17.794350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:17.794320    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:19.854526 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:19.854483    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:19.866769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:19.866746    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:19.980612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:19.980562    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4" podStartSLOduration=10.980550458 podStartE2EDuration="10.980550458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:10.721456927 +0000 UTC m=+1300.411340054" watchObservedRunningTime="2026-10-31 01:18:19.980550458 +0000 UTC m=+1309.670433585"
Oct 31 01:18:20.667798 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:20.667769    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:18:21.486930 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:21.486884    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:26.275672 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:26.275567    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:27.160702 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.160671    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"]
Oct 31 01:18:27.164753 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.164728    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.269708 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.269657    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.269864 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.269734    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5245c2fd-f28f-4965-a210-bc0cf9a10376-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.269864 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.269793    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.269941 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.269869    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.269975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.269942    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.270022 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.270007    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.270060 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.270036    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.270217 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.270199    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxfvf\" (UniqueName: \"kubernetes.io/projected/5245c2fd-f28f-4965-a210-bc0cf9a10376-kube-api-access-cxfvf\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.270256 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.270243    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.360337 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.360302    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"]
Oct 31 01:18:27.371292 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.371265    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxfvf\" (UniqueName: \"kubernetes.io/projected/5245c2fd-f28f-4965-a210-bc0cf9a10376-kube-api-access-cxfvf\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.371462 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.371315    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.371462 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.371358    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.371462 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.371385    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5245c2fd-f28f-4965-a210-bc0cf9a10376-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.371462 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.371428    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.371697 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.371504    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.371697 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.371575    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.371697 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.371638    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.371697 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.371653    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.371993 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.371971    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.372084 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.372059    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.372210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.372161    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.372395 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.372374    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.372856 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.372831    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5245c2fd-f28f-4965-a210-bc0cf9a10376-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.374012 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.373984    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.374479 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.374457    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.427228 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.427133    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.427355 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.427339    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxfvf\" (UniqueName: \"kubernetes.io/projected/5245c2fd-f28f-4965-a210-bc0cf9a10376-kube-api-access-cxfvf\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.478900 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.478865    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:27.916591 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.916564    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"]
Oct 31 01:18:27.916953 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:18:27.916932    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5245c2fd_f28f_4965_a210_bc0cf9a10376.slice/crio-c0a4a309ebad08fe63be732f2490fa45c393b59717ece72f6287f3534efb4db0 WatchSource:0}: Error finding container c0a4a309ebad08fe63be732f2490fa45c393b59717ece72f6287f3534efb4db0: Status 404 returned error can't find the container with id c0a4a309ebad08fe63be732f2490fa45c393b59717ece72f6287f3534efb4db0
Oct 31 01:18:27.919855 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.919809    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:18:27.920132 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.920101    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:18:27.920341 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:27.920315    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:18:28.704508 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:28.704468    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg" event={"ID":"5245c2fd-f28f-4965-a210-bc0cf9a10376","Type":"ContainerStarted","Data":"d8fa899b917948c742825f40c6da2bd3c5f03926bb8c5b14df82dcc805e77bf0"}
Oct 31 01:18:28.705010 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:28.704540    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg" event={"ID":"5245c2fd-f28f-4965-a210-bc0cf9a10376","Type":"ContainerStarted","Data":"c0a4a309ebad08fe63be732f2490fa45c393b59717ece72f6287f3534efb4db0"}
Oct 31 01:18:28.925271 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:28.925215    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg" podStartSLOduration=1.92520169 podStartE2EDuration="1.92520169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:28.874666864 +0000 UTC m=+1318.564550004" watchObservedRunningTime="2026-10-31 01:18:28.92520169 +0000 UTC m=+1318.615084816"
Oct 31 01:18:29.205553 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:18:29.205515    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:18:29.483252 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:29.482211    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:29.483252 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:29.482265    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 01:18:29.483252 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:29.482308    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg" podUID="5245c2fd-f28f-4965-a210-bc0cf9a10376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 01:18:29.497422 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:29.497392    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"]
Oct 31 01:18:29.585552 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:29.585521    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:30.484611 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:30.484574    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 01:18:30.485038 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:30.484632    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg" podUID="5245c2fd-f28f-4965-a210-bc0cf9a10376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 01:18:31.091607 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:31.091574    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:31.484816 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:31.484731    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 01:18:31.484816 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:31.484797    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg" podUID="5245c2fd-f28f-4965-a210-bc0cf9a10376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 01:18:31.606780 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:31.606734    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:32.385253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:32.385218    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:32.482202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:32.479240    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 01:18:32.482202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:32.479301    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg" podUID="5245c2fd-f28f-4965-a210-bc0cf9a10376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 01:18:32.555850 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:32.555815    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:32.741787 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:32.741675    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:33.482200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:33.481602    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 01:18:33.482200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:33.481663    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg" podUID="5245c2fd-f28f-4965-a210-bc0cf9a10376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 01:18:34.479738 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:34.479707    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 01:18:34.480131 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:34.479760    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg" podUID="5245c2fd-f28f-4965-a210-bc0cf9a10376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 01:18:35.483931 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:35.483903    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:35.484391 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:35.484372    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:35.485017 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:35.484999    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:18:36.068955 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:36.068920    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:41.068687 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:41.068650    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:44.521520 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.521460    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c" podUID="219c8fed-42af-45a6-a2d9-e95a794995a4" containerName="main" containerID="cri-o://d8d03ac8dcb292434966074ed180c3a61ccea80e1454a4ee17a1bf97d9f15310" gracePeriod=45
Oct 31 01:18:44.784068 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.784046    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:18:44.855294 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.855263    4476 generic.go:356] "Generic (PLEG): container finished" podID="219c8fed-42af-45a6-a2d9-e95a794995a4" containerID="d8d03ac8dcb292434966074ed180c3a61ccea80e1454a4ee17a1bf97d9f15310" exitCode=0
Oct 31 01:18:44.855442 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.855312    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"
Oct 31 01:18:44.855442 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.855353    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c" event={"ID":"219c8fed-42af-45a6-a2d9-e95a794995a4","Type":"ContainerDied","Data":"d8d03ac8dcb292434966074ed180c3a61ccea80e1454a4ee17a1bf97d9f15310"}
Oct 31 01:18:44.855442 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.855395    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c" event={"ID":"219c8fed-42af-45a6-a2d9-e95a794995a4","Type":"ContainerDied","Data":"09f992b262e234bcfd4bfca966d8c5f45fb294919b9e90d7b14913ca24d08189"}
Oct 31 01:18:44.855442 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.855418    4476 scope.go:122] "RemoveContainer" containerID="d8d03ac8dcb292434966074ed180c3a61ccea80e1454a4ee17a1bf97d9f15310"
Oct 31 01:18:44.864351 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.864336    4476 scope.go:122] "RemoveContainer" containerID="d8d03ac8dcb292434966074ed180c3a61ccea80e1454a4ee17a1bf97d9f15310"
Oct 31 01:18:44.864616 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:18:44.864597    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d03ac8dcb292434966074ed180c3a61ccea80e1454a4ee17a1bf97d9f15310\": container with ID starting with d8d03ac8dcb292434966074ed180c3a61ccea80e1454a4ee17a1bf97d9f15310 not found: ID does not exist" containerID="d8d03ac8dcb292434966074ed180c3a61ccea80e1454a4ee17a1bf97d9f15310"
Oct 31 01:18:44.864667 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.864622    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d03ac8dcb292434966074ed180c3a61ccea80e1454a4ee17a1bf97d9f15310"} err="failed to get container status \"d8d03ac8dcb292434966074ed180c3a61ccea80e1454a4ee17a1bf97d9f15310\": rpc error: code = NotFound desc = could not find container \"d8d03ac8dcb292434966074ed180c3a61ccea80e1454a4ee17a1bf97d9f15310\": container with ID starting with d8d03ac8dcb292434966074ed180c3a61ccea80e1454a4ee17a1bf97d9f15310 not found: ID does not exist"
Oct 31 01:18:44.925889 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.925862    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-model-cache\") pod \"219c8fed-42af-45a6-a2d9-e95a794995a4\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") "
Oct 31 01:18:44.926052 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.925922    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-dshm\") pod \"219c8fed-42af-45a6-a2d9-e95a794995a4\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") "
Oct 31 01:18:44.926052 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.925951    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-home\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-home\") pod \"219c8fed-42af-45a6-a2d9-e95a794995a4\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") "
Oct 31 01:18:44.926052 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.925995    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-tmp-dir\") pod \"219c8fed-42af-45a6-a2d9-e95a794995a4\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") "
Oct 31 01:18:44.926268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.926059    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219c8fed-42af-45a6-a2d9-e95a794995a4-kube-api-access-h7m6l\" (UniqueName: \"kubernetes.io/projected/219c8fed-42af-45a6-a2d9-e95a794995a4-kube-api-access-h7m6l\") pod \"219c8fed-42af-45a6-a2d9-e95a794995a4\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") "
Oct 31 01:18:44.926268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.926149    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-model-cache" pod "219c8fed-42af-45a6-a2d9-e95a794995a4" (UID: "219c8fed-42af-45a6-a2d9-e95a794995a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:18:44.926268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.926258    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/219c8fed-42af-45a6-a2d9-e95a794995a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/219c8fed-42af-45a6-a2d9-e95a794995a4-tls-certs\") pod \"219c8fed-42af-45a6-a2d9-e95a794995a4\" (UID: \"219c8fed-42af-45a6-a2d9-e95a794995a4\") "
Oct 31 01:18:44.926412 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.926318    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-home" pod "219c8fed-42af-45a6-a2d9-e95a794995a4" (UID: "219c8fed-42af-45a6-a2d9-e95a794995a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:18:44.926412 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.926401    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-tmp-dir" pod "219c8fed-42af-45a6-a2d9-e95a794995a4" (UID: "219c8fed-42af-45a6-a2d9-e95a794995a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:18:44.926752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.926725    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:44.926827 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.926760    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:44.926827 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.926775    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:44.929288 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.929261    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-dshm" pod "219c8fed-42af-45a6-a2d9-e95a794995a4" (UID: "219c8fed-42af-45a6-a2d9-e95a794995a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:18:44.929403 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.929289    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219c8fed-42af-45a6-a2d9-e95a794995a4-tls-certs" pod "219c8fed-42af-45a6-a2d9-e95a794995a4" (UID: "219c8fed-42af-45a6-a2d9-e95a794995a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:18:44.929403 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:44.929309    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219c8fed-42af-45a6-a2d9-e95a794995a4-kube-api-access-h7m6l" pod "219c8fed-42af-45a6-a2d9-e95a794995a4" (UID: "219c8fed-42af-45a6-a2d9-e95a794995a4"). InnerVolumeSpecName "kube-api-access-h7m6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:18:45.027563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:45.027538    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/219c8fed-42af-45a6-a2d9-e95a794995a4-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:45.027563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:45.027559    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/219c8fed-42af-45a6-a2d9-e95a794995a4-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:45.027756 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:45.027569    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7m6l\" (UniqueName: \"kubernetes.io/projected/219c8fed-42af-45a6-a2d9-e95a794995a4-kube-api-access-h7m6l\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:45.185078 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:18:45.185048    4476 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod219c8fed_42af_45a6_a2d9_e95a794995a4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod219c8fed_42af_45a6_a2d9_e95a794995a4.slice/crio-09f992b262e234bcfd4bfca966d8c5f45fb294919b9e90d7b14913ca24d08189\": RecentStats: unable to find data in memory cache]"
Oct 31 01:18:45.274852 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:45.274824    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"]
Oct 31 01:18:45.324352 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:45.324327    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/e2e-conflict-b-c17e7941-kserve-5bbb4997c6-jsj8c"]
Oct 31 01:18:45.817605 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:45.817577    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:45.997454 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:45.997419    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:46.453793 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:46.453758    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="219c8fed-42af-45a6-a2d9-e95a794995a4" path="/var/lib/kubelet/pods/219c8fed-42af-45a6-a2d9-e95a794995a4/volumes"
Oct 31 01:18:47.160998 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:47.160965    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"]
Oct 31 01:18:50.488293 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:50.488258    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:52.161369 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.161331    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg" podUID="cefbd225-13da-4085-9088-865d4b331e3d" containerName="payload-processing" containerID="cri-o://bbc965c9ccbdf14b9073a71b134df9e07267af63d4d7fd14d05fd6667eeb0b99" gracePeriod=25
Oct 31 01:18:52.431257 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.431233    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:18:52.509085 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.509062    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cefbd225-13da-4085-9088-865d4b331e3d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cefbd225-13da-4085-9088-865d4b331e3d-routing-overlay-volume\") pod \"cefbd225-13da-4085-9088-865d4b331e3d\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") "
Oct 31 01:18:52.509246 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.509116    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials\") pod \"cefbd225-13da-4085-9088-865d4b331e3d\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") "
Oct 31 01:18:52.509246 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.509226    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-kube-api-access-gm5tx\" (UniqueName: \"kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-kube-api-access-gm5tx\") pod \"cefbd225-13da-4085-9088-865d4b331e3d\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") "
Oct 31 01:18:52.509335 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.509296    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cefbd225-13da-4085-9088-865d4b331e3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefbd225-13da-4085-9088-865d4b331e3d-plugins-config-volume\") pod \"cefbd225-13da-4085-9088-865d4b331e3d\" (UID: \"cefbd225-13da-4085-9088-865d4b331e3d\") "
Oct 31 01:18:52.509440 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.509415    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cefbd225-13da-4085-9088-865d4b331e3d-routing-overlay-volume" pod "cefbd225-13da-4085-9088-865d4b331e3d" (UID: "cefbd225-13da-4085-9088-865d4b331e3d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:18:52.509716 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.509696    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cefbd225-13da-4085-9088-865d4b331e3d-plugins-config-volume" pod "cefbd225-13da-4085-9088-865d4b331e3d" (UID: "cefbd225-13da-4085-9088-865d4b331e3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:18:52.509790 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.509706    4476 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cefbd225-13da-4085-9088-865d4b331e3d-routing-overlay-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:52.511511 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.511487    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials" pod "cefbd225-13da-4085-9088-865d4b331e3d" (UID: "cefbd225-13da-4085-9088-865d4b331e3d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:18:52.511511 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.511502    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-kube-api-access-gm5tx" pod "cefbd225-13da-4085-9088-865d4b331e3d" (UID: "cefbd225-13da-4085-9088-865d4b331e3d"). InnerVolumeSpecName "kube-api-access-gm5tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:18:52.610935 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.610907    4476 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-provider-credentials\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:52.610935 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.610933    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm5tx\" (UniqueName: \"kubernetes.io/projected/cefbd225-13da-4085-9088-865d4b331e3d-kube-api-access-gm5tx\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:52.610935 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.610943    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefbd225-13da-4085-9088-865d4b331e3d-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:52.904327 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.904293    4476 generic.go:356] "Generic (PLEG): container finished" podID="cefbd225-13da-4085-9088-865d4b331e3d" containerID="bbc965c9ccbdf14b9073a71b134df9e07267af63d4d7fd14d05fd6667eeb0b99" exitCode=0
Oct 31 01:18:52.904494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.904341    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"
Oct 31 01:18:52.904494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.904369    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg" event={"ID":"cefbd225-13da-4085-9088-865d4b331e3d","Type":"ContainerDied","Data":"bbc965c9ccbdf14b9073a71b134df9e07267af63d4d7fd14d05fd6667eeb0b99"}
Oct 31 01:18:52.904494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.904428    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg" event={"ID":"cefbd225-13da-4085-9088-865d4b331e3d","Type":"ContainerDied","Data":"7951e1f98a20bb08ead9e59536dbb39a0528e19dce537d87e979f9d36da249c9"}
Oct 31 01:18:52.904494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.904458    4476 scope.go:122] "RemoveContainer" containerID="bbc965c9ccbdf14b9073a71b134df9e07267af63d4d7fd14d05fd6667eeb0b99"
Oct 31 01:18:52.914267 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.914246    4476 scope.go:122] "RemoveContainer" containerID="bbc965c9ccbdf14b9073a71b134df9e07267af63d4d7fd14d05fd6667eeb0b99"
Oct 31 01:18:52.914520 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:18:52.914501    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc965c9ccbdf14b9073a71b134df9e07267af63d4d7fd14d05fd6667eeb0b99\": container with ID starting with bbc965c9ccbdf14b9073a71b134df9e07267af63d4d7fd14d05fd6667eeb0b99 not found: ID does not exist" containerID="bbc965c9ccbdf14b9073a71b134df9e07267af63d4d7fd14d05fd6667eeb0b99"
Oct 31 01:18:52.914606 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:52.914528    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc965c9ccbdf14b9073a71b134df9e07267af63d4d7fd14d05fd6667eeb0b99"} err="failed to get container status \"bbc965c9ccbdf14b9073a71b134df9e07267af63d4d7fd14d05fd6667eeb0b99\": rpc error: code = NotFound desc = could not find container \"bbc965c9ccbdf14b9073a71b134df9e07267af63d4d7fd14d05fd6667eeb0b99\": container with ID starting with bbc965c9ccbdf14b9073a71b134df9e07267af63d4d7fd14d05fd6667eeb0b99 not found: ID does not exist"
Oct 31 01:18:53.068509 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:53.068477    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"]
Oct 31 01:18:53.117794 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:53.117768    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fb874b485-9dxqg"]
Oct 31 01:18:54.454606 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:54.454571    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefbd225-13da-4085-9088-865d4b331e3d" path="/var/lib/kubelet/pods/cefbd225-13da-4085-9088-865d4b331e3d/volumes"
Oct 31 01:18:55.267394 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:55.267365    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:58.667263 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.667232    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"]
Oct 31 01:18:58.667759 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.667711    4476 cpu_manager.go:414] "containerMap: removing container" podUID="cefbd225-13da-4085-9088-865d4b331e3d" containerName="payload-processing"
Oct 31 01:18:58.667759 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.667722    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefbd225-13da-4085-9088-865d4b331e3d" containerName="payload-processing"
Oct 31 01:18:58.667759 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.667757    4476 cpu_manager.go:414] "containerMap: removing container" podUID="219c8fed-42af-45a6-a2d9-e95a794995a4" containerName="main"
Oct 31 01:18:58.667912 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.667762    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219c8fed-42af-45a6-a2d9-e95a794995a4" containerName="main"
Oct 31 01:18:58.667912 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.667831    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="219c8fed-42af-45a6-a2d9-e95a794995a4" containerName="main"
Oct 31 01:18:58.667912 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.667841    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefbd225-13da-4085-9088-865d4b331e3d" containerName="payload-processing"
Oct 31 01:18:58.672569 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.672551    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.779498 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.779465    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"]
Oct 31 01:18:58.781223 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.781162    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.781318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.781288    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97eb81c-138f-42d7-a47e-a19504d68e7c-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.781377 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.781338    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfmxf\" (UniqueName: \"kubernetes.io/projected/a97eb81c-138f-42d7-a47e-a19504d68e7c-kube-api-access-qfmxf\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.781476 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.781463    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.781533 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.781496    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.781588 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.781535    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.781742 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.781713    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.781788 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.781758    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.781874 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.781854    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.883909 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.883856    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfmxf\" (UniqueName: \"kubernetes.io/projected/a97eb81c-138f-42d7-a47e-a19504d68e7c-kube-api-access-qfmxf\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.884129 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.884026    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.884129 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.884086    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.884298 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.884134    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.884298 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.884237    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.884298 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.884267    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.884444 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.884300    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.884444 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.884376    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.884549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.884453    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97eb81c-138f-42d7-a47e-a19504d68e7c-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.884678 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.884653    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.885358 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.885078    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.885358 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.885346    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97eb81c-138f-42d7-a47e-a19504d68e7c-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.885742 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.885418    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.885742 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.885661    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.887378 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.887347    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.887512 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.887489    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.941344 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.941272    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.941517 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.941475    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfmxf\" (UniqueName: \"kubernetes.io/projected/a97eb81c-138f-42d7-a47e-a19504d68e7c-kube-api-access-qfmxf\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:58.985317 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:58.985278    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:59.188097 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:18:59.188053    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:18:59.425845 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:59.425814    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"]
Oct 31 01:18:59.427782 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:18:59.427755    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97eb81c_138f_42d7_a47e_a19504d68e7c.slice/crio-77a310f660e454a676574758c6328162a2ed305e28efe956c0ee9c8fd91a67a0 WatchSource:0}: Error finding container 77a310f660e454a676574758c6328162a2ed305e28efe956c0ee9c8fd91a67a0: Status 404 returned error can't find the container with id 77a310f660e454a676574758c6328162a2ed305e28efe956c0ee9c8fd91a67a0
Oct 31 01:18:59.430957 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:59.430912    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:18:59.431078 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:59.431010    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:18:59.431078 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:59.431057    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:18:59.974201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:59.962589    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn" event={"ID":"a97eb81c-138f-42d7-a47e-a19504d68e7c","Type":"ContainerStarted","Data":"512047f077b2b2a616a05c7cea5f61043cddf37c2e03a997234b67031f968ef9"}
Oct 31 01:18:59.974201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:59.962699    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn" event={"ID":"a97eb81c-138f-42d7-a47e-a19504d68e7c","Type":"ContainerStarted","Data":"77a310f660e454a676574758c6328162a2ed305e28efe956c0ee9c8fd91a67a0"}
Oct 31 01:18:59.999198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:59.987264    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:18:59.999198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:18:59.990541    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:00.130648 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:00.130583    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn" podStartSLOduration=2.130560204 podStartE2EDuration="2.130560204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:00.101692727 +0000 UTC m=+1349.791575854" watchObservedRunningTime="2026-10-31 01:19:00.130560204 +0000 UTC m=+1349.820443332"
Oct 31 01:19:01.048204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:01.045123    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:19:01.048204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:01.045201    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn" podUID="a97eb81c-138f-42d7-a47e-a19504d68e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 01:19:01.986436 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:01.986390    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:19:01.986617 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:01.986472    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn" podUID="a97eb81c-138f-42d7-a47e-a19504d68e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 01:19:02.925166 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:02.925134    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:02.985710 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:02.985676    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:19:02.985900 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:02.985739    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn" podUID="a97eb81c-138f-42d7-a47e-a19504d68e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 01:19:03.592402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:03.592371    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"]
Oct 31 01:19:03.659153 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:03.659126    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:03.985694 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:03.985598    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:19:03.985694 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:03.985656    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn" podUID="a97eb81c-138f-42d7-a47e-a19504d68e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 01:19:04.985683 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:04.985645    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:19:04.985892 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:04.985719    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn" podUID="a97eb81c-138f-42d7-a47e-a19504d68e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 01:19:05.112149 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:05.112101    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:05.985955 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:05.985919    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:19:05.986401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:05.985975    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn" podUID="a97eb81c-138f-42d7-a47e-a19504d68e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 01:19:06.352734 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:06.352695    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:06.990527 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:06.990498    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:19:06.990986 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:06.990786    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:19:06.991512 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:06.991496    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:19:08.263325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:08.263295    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:09.889144 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:09.889110    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:15.009320 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:15.009280    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:18.613078 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.613015    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt" podUID="2ea8587f-d236-49e3-ae23-24ad6ad3a366" containerName="main" containerID="cri-o://caba281289f9fbfa03d7faf32de3ecf0b4685820de2304e68d630a67d574b4a6" gracePeriod=45
Oct 31 01:19:18.865122 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.865062    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:19:18.883469 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.883445    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ea8587f-d236-49e3-ae23-24ad6ad3a366-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea8587f-d236-49e3-ae23-24ad6ad3a366-tls-certs\") pod \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") "
Oct 31 01:19:18.883582 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.883489    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-dshm\") pod \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") "
Oct 31 01:19:18.883582 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.883531    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-tmp-dir\") pod \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") "
Oct 31 01:19:18.883582 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.883559    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea8587f-d236-49e3-ae23-24ad6ad3a366-kube-api-access-j7b7q\" (UniqueName: \"kubernetes.io/projected/2ea8587f-d236-49e3-ae23-24ad6ad3a366-kube-api-access-j7b7q\") pod \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") "
Oct 31 01:19:18.883723 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.883591    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-home\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-home\") pod \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") "
Oct 31 01:19:18.883723 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.883695    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-model-cache\") pod \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\" (UID: \"2ea8587f-d236-49e3-ae23-24ad6ad3a366\") "
Oct 31 01:19:18.883831 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.883809    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-tmp-dir" pod "2ea8587f-d236-49e3-ae23-24ad6ad3a366" (UID: "2ea8587f-d236-49e3-ae23-24ad6ad3a366"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:18.883962 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.883933    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-home" pod "2ea8587f-d236-49e3-ae23-24ad6ad3a366" (UID: "2ea8587f-d236-49e3-ae23-24ad6ad3a366"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:18.884038 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.883964    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-model-cache" pod "2ea8587f-d236-49e3-ae23-24ad6ad3a366" (UID: "2ea8587f-d236-49e3-ae23-24ad6ad3a366"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:18.884225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.884189    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:18.884225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.884208    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:18.884225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.884217    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:18.885858 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.885832    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea8587f-d236-49e3-ae23-24ad6ad3a366-kube-api-access-j7b7q" pod "2ea8587f-d236-49e3-ae23-24ad6ad3a366" (UID: "2ea8587f-d236-49e3-ae23-24ad6ad3a366"). InnerVolumeSpecName "kube-api-access-j7b7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:18.886319 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.886300    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea8587f-d236-49e3-ae23-24ad6ad3a366-tls-certs" pod "2ea8587f-d236-49e3-ae23-24ad6ad3a366" (UID: "2ea8587f-d236-49e3-ae23-24ad6ad3a366"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:19:18.886319 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.886304    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-dshm" pod "2ea8587f-d236-49e3-ae23-24ad6ad3a366" (UID: "2ea8587f-d236-49e3-ae23-24ad6ad3a366"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:18.985185 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.985148    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea8587f-d236-49e3-ae23-24ad6ad3a366-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:18.985313 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.985200    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea8587f-d236-49e3-ae23-24ad6ad3a366-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:18.985313 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:18.985218    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7b7q\" (UniqueName: \"kubernetes.io/projected/2ea8587f-d236-49e3-ae23-24ad6ad3a366-kube-api-access-j7b7q\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:19.378819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:19.378786    4476 generic.go:356] "Generic (PLEG): container finished" podID="2ea8587f-d236-49e3-ae23-24ad6ad3a366" containerID="caba281289f9fbfa03d7faf32de3ecf0b4685820de2304e68d630a67d574b4a6" exitCode=0
Oct 31 01:19:19.378984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:19.378837    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"
Oct 31 01:19:19.378984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:19.378850    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt" event={"ID":"2ea8587f-d236-49e3-ae23-24ad6ad3a366","Type":"ContainerDied","Data":"caba281289f9fbfa03d7faf32de3ecf0b4685820de2304e68d630a67d574b4a6"}
Oct 31 01:19:19.378984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:19.378973    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt" event={"ID":"2ea8587f-d236-49e3-ae23-24ad6ad3a366","Type":"ContainerDied","Data":"a6c32643039fe45a0f7d6d72555dc4416eaaa988ac3194adb2c93d550e6d7120"}
Oct 31 01:19:19.379087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:19.378998    4476 scope.go:122] "RemoveContainer" containerID="caba281289f9fbfa03d7faf32de3ecf0b4685820de2304e68d630a67d574b4a6"
Oct 31 01:19:19.388681 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:19.388663    4476 scope.go:122] "RemoveContainer" containerID="caba281289f9fbfa03d7faf32de3ecf0b4685820de2304e68d630a67d574b4a6"
Oct 31 01:19:19.388978 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:19:19.388954    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caba281289f9fbfa03d7faf32de3ecf0b4685820de2304e68d630a67d574b4a6\": container with ID starting with caba281289f9fbfa03d7faf32de3ecf0b4685820de2304e68d630a67d574b4a6 not found: ID does not exist" containerID="caba281289f9fbfa03d7faf32de3ecf0b4685820de2304e68d630a67d574b4a6"
Oct 31 01:19:19.389044 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:19.388986    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caba281289f9fbfa03d7faf32de3ecf0b4685820de2304e68d630a67d574b4a6"} err="failed to get container status \"caba281289f9fbfa03d7faf32de3ecf0b4685820de2304e68d630a67d574b4a6\": rpc error: code = NotFound desc = could not find container \"caba281289f9fbfa03d7faf32de3ecf0b4685820de2304e68d630a67d574b4a6\": container with ID starting with caba281289f9fbfa03d7faf32de3ecf0b4685820de2304e68d630a67d574b4a6 not found: ID does not exist"
Oct 31 01:19:19.540577 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:19.540542    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"]
Oct 31 01:19:19.543723 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:19.543698    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt"]
Oct 31 01:19:19.638564 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:19.638491    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:19.825563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:19.825535    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:19.828483 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:19.828443    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ec695b/e2e-conflict-a-c17e7941-kserve-7f8f4b4b58-fsdqt" podUID="2ea8587f-d236-49e3-ae23-24ad6ad3a366" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 01:19:20.454194 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:20.454140    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea8587f-d236-49e3-ae23-24ad6ad3a366" path="/var/lib/kubelet/pods/2ea8587f-d236-49e3-ae23-24ad6ad3a366/volumes"
Oct 31 01:19:23.946489 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:23.946452    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"]
Oct 31 01:19:23.946972 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:23.946927    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2ea8587f-d236-49e3-ae23-24ad6ad3a366" containerName="main"
Oct 31 01:19:23.946972 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:23.946937    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea8587f-d236-49e3-ae23-24ad6ad3a366" containerName="main"
Oct 31 01:19:23.947043 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:23.947033    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea8587f-d236-49e3-ae23-24ad6ad3a366" containerName="main"
Oct 31 01:19:23.951158 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:23.951142    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.038034 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.037998    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-model-cache\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.038227 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.038118    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-home\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.038349 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.038328    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdl8q\" (UniqueName: \"kubernetes.io/projected/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-kube-api-access-cdl8q\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.038396 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.038378    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-tmp-dir\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.038475 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.038460    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-tls-certs\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.038553 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.038539    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-dshm\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.052838 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.052811    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-acd385\"/\"distinct-2-w4-acd385-kserve-self-signed-certs\""
Oct 31 01:19:24.055969 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.055945    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"]
Oct 31 01:19:24.091611 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.091451    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:24.139880 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.139837    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdl8q\" (UniqueName: \"kubernetes.io/projected/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-kube-api-access-cdl8q\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.140191 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.140148    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-tmp-dir\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.140322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.140235    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-tls-certs\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.140322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.140312    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-dshm\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.140435 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.140345    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-model-cache\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.140435 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.140413    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-home\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.140882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.140859    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-home\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.141029 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.140939    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-model-cache\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.141162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.141055    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-tmp-dir\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.143802 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.143763    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-tls-certs\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.143952 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.143937    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-dshm\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.197634 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.197547    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdl8q\" (UniqueName: \"kubernetes.io/projected/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-kube-api-access-cdl8q\") pod \"distinct-2-w4-acd385-kserve-66b5f46688-gb9bx\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.263495 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.263463    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:24.496256 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.496116    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:24.646057 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:24.646027    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"]
Oct 31 01:19:24.647637 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:19:24.647611    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a67d783_3ecc_4425_8e50_8b1e72f4d57b.slice/crio-adb2387948723e3927bd54d5a63363a92b83aa65c34ced56edee4cb1f47b68bc WatchSource:0}: Error finding container adb2387948723e3927bd54d5a63363a92b83aa65c34ced56edee4cb1f47b68bc: Status 404 returned error can't find the container with id adb2387948723e3927bd54d5a63363a92b83aa65c34ced56edee4cb1f47b68bc
Oct 31 01:19:25.416947 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:25.416913    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" event={"ID":"4a67d783-3ecc-4425-8e50-8b1e72f4d57b","Type":"ContainerStarted","Data":"de0a692e3e09cccb263520487ccb5d3ca80a23ad26d39185f4619252b0c6a85c"}
Oct 31 01:19:25.416947 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:25.416955    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" event={"ID":"4a67d783-3ecc-4425-8e50-8b1e72f4d57b","Type":"ContainerStarted","Data":"adb2387948723e3927bd54d5a63363a92b83aa65c34ced56edee4cb1f47b68bc"}
Oct 31 01:19:25.534992 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:25.534941    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" podStartSLOduration=2.534925796 podStartE2EDuration="2.534925796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:25.532638277 +0000 UTC m=+1375.222521404" watchObservedRunningTime="2026-10-31 01:19:25.534925796 +0000 UTC m=+1375.224808955"
Oct 31 01:19:26.264409 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:26.264380    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:26.529299 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:26.529267    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:27.641962 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.641934    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"]
Oct 31 01:19:27.645832 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.645815    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.697675 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.697657    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e621db-openshift-default-dockercfg-p8kgb\""
Oct 31 01:19:27.749937 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.749912    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"]
Oct 31 01:19:27.798605 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.798578    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.798750 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.798627    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.798750 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.798709    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.798822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.798748    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.798822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.798769    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vhfp\" (UniqueName: \"kubernetes.io/projected/2a84b045-58f1-43bf-b02b-6ee4c9a62725-kube-api-access-2vhfp\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.798933 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.798843    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.798970 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.798943    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.799004 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.798982    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.799034 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.799015    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.900609 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.900537    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.900732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.900608    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.900772 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.900745    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.900821 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.900806    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.900882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.900867    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.900920 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.900895    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.900964 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.900938    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.900964 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.900948    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.901033 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.900973    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vhfp\" (UniqueName: \"kubernetes.io/projected/2a84b045-58f1-43bf-b02b-6ee4c9a62725-kube-api-access-2vhfp\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.901033 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.900995    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.901144 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.901044    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.901144 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.901093    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.901501 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.901478    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.901593 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.901533    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.903325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.903303    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.903434 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.903420    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.957298 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.957273    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vhfp\" (UniqueName: \"kubernetes.io/projected/2a84b045-58f1-43bf-b02b-6ee4c9a62725-kube-api-access-2vhfp\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.957551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.957535    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:27.959561 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:27.959538    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:28.390353 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:28.390327    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"]
Oct 31 01:19:28.391456 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:19:28.391427    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a84b045_58f1_43bf_b02b_6ee4c9a62725.slice/crio-4e309f5e1eef760941902736aa8846b42d3134d114db2357ee22f81752fab4d6 WatchSource:0}: Error finding container 4e309f5e1eef760941902736aa8846b42d3134d114db2357ee22f81752fab4d6: Status 404 returned error can't find the container with id 4e309f5e1eef760941902736aa8846b42d3134d114db2357ee22f81752fab4d6
Oct 31 01:19:28.393940 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:28.393898    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:19:28.394018 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:28.393973    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:19:28.394018 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:28.394004    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:19:28.411012 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:28.410985    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:28.436606 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:28.436559    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7" event={"ID":"2a84b045-58f1-43bf-b02b-6ee4c9a62725","Type":"ContainerStarted","Data":"4e309f5e1eef760941902736aa8846b42d3134d114db2357ee22f81752fab4d6"}
Oct 31 01:19:29.189584 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:29.186620    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:29.211353 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:19:29.211316    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:19:29.444802 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:29.444713    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7" event={"ID":"2a84b045-58f1-43bf-b02b-6ee4c9a62725","Type":"ContainerStarted","Data":"bb14bd08eb6fa6ae42de87d7d657a2e25aa5dc27f59777916da8b7367a7200d1"}
Oct 31 01:19:29.613262 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:29.613199    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7" podStartSLOduration=2.613166395 podStartE2EDuration="2.613166395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:29.611729212 +0000 UTC m=+1379.301612340" watchObservedRunningTime="2026-10-31 01:19:29.613166395 +0000 UTC m=+1379.303049522"
Oct 31 01:19:29.981201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:29.965573    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:29.981201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:29.974419    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:19:29.981201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:29.974474    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7" podUID="2a84b045-58f1-43bf-b02b-6ee4c9a62725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:19:30.984689 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:30.984584    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:19:30.984689 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:30.984643    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7" podUID="2a84b045-58f1-43bf-b02b-6ee4c9a62725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:19:31.974728 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:31.974695    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:19:31.975044 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:31.975003    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7" podUID="2a84b045-58f1-43bf-b02b-6ee4c9a62725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:19:32.912750 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:32.912715    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"]
Oct 31 01:19:32.918538 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:32.918515    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:32.987202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:32.985056    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:19:32.987202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:32.985114    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7" podUID="2a84b045-58f1-43bf-b02b-6ee4c9a62725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:19:32.991196 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:32.990348    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-dshm\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:32.991196 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:32.990413    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-tmp-dir\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:32.991196 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:32.990474    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-model-cache\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:32.991196 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:32.990514    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/912dfe65-0c1e-4445-9d00-cf197ad4a93b-tls-certs\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:32.991196 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:32.990667    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-home\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:32.991196 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:32.990712    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sv6h\" (UniqueName: \"kubernetes.io/projected/912dfe65-0c1e-4445-9d00-cf197ad4a93b-kube-api-access-4sv6h\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:32.991196 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:32.990901    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"]
Oct 31 01:19:33.022139 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.022103    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ec695b\"/\"kube-root-ca.crt\""
Oct 31 01:19:33.022311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.022236    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ec695b\"/\"openshift-service-ca.crt\""
Oct 31 01:19:33.022311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.022108    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ec695b\"/\"default-dockercfg-87kps\""
Oct 31 01:19:33.022615 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.022592    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ec695b\"/\"distinct-w3-ec695b-kserve-self-signed-certs\""
Oct 31 01:19:33.080913 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.080880    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:33.101339 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.096075    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/912dfe65-0c1e-4445-9d00-cf197ad4a93b-tls-certs\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:33.101701 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.101682    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-home\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:33.101857 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.101843    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sv6h\" (UniqueName: \"kubernetes.io/projected/912dfe65-0c1e-4445-9d00-cf197ad4a93b-kube-api-access-4sv6h\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:33.102093 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.102077    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-dshm\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:33.102249 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.102235    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-tmp-dir\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:33.105422 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.105396    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-model-cache\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:33.105740 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.104383    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-tmp-dir\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:33.105874 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.102780    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-home\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:33.106397 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.106374    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-model-cache\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:33.119631 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.119311    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-dshm\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:33.120675 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.120312    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/912dfe65-0c1e-4445-9d00-cf197ad4a93b-tls-certs\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:33.176988 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.176906    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sv6h\" (UniqueName: \"kubernetes.io/projected/912dfe65-0c1e-4445-9d00-cf197ad4a93b-kube-api-access-4sv6h\") pod \"distinct-w3-ec695b-kserve-669f64756-fr467\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:33.232812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.232777    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:33.700200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.699934    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"]
Oct 31 01:19:33.956535 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.956501    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:33.964198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.960741    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:19:33.964198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:33.960790    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7" podUID="2a84b045-58f1-43bf-b02b-6ee4c9a62725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:19:34.280253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:34.269259    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:34.423199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:34.422545    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:34.557311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:34.557224    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" event={"ID":"912dfe65-0c1e-4445-9d00-cf197ad4a93b","Type":"ContainerStarted","Data":"8e854702261c76f9b0cd85ef7662939f074a99d4a7f996f3205f42b9d617ab28"}
Oct 31 01:19:34.557311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:34.557273    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" event={"ID":"912dfe65-0c1e-4445-9d00-cf197ad4a93b","Type":"ContainerStarted","Data":"316e97c9b606c2c2aca3b1584e753cd8c506eb04663a711dd6d96b8938ef403f"}
Oct 31 01:19:34.587199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:34.579432    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:34.683203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:34.681308    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" podStartSLOduration=2.68128943 podStartE2EDuration="2.68128943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:34.680560837 +0000 UTC m=+1384.370443965" watchObservedRunningTime="2026-10-31 01:19:34.68128943 +0000 UTC m=+1384.371172557"
Oct 31 01:19:34.981694 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:34.981603    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:19:35.009399 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:35.009357    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7" podUID="2a84b045-58f1-43bf-b02b-6ee4c9a62725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:19:35.233620 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:35.233521    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:36.045097 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:36.045067    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:36.045621 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:36.045574    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:36.046515 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:36.046494    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:19:38.958187 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:38.958133    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:41.483907 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.483829    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"]
Oct 31 01:19:41.487833 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.487816    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.546197 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.545502    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-97b317-openshift-default-dockercfg-4lml6\""
Oct 31 01:19:41.554505 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.554472    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.554655 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.554520    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.554655 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.554556    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9rsh\" (UniqueName: \"kubernetes.io/projected/8e2f4139-a07c-4159-adb4-8fbad0fe1897-kube-api-access-x9rsh\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.554655 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.554579    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.554888 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.554657    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.554888 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.554680    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.554888 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.554697    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.554888 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.554726    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.554888 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.554772    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.608678 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.608517    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"]
Oct 31 01:19:41.655988 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.655960    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.656146 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.655999    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.656146 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.656017    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.656146 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.656046    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.656146 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.656085    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.656146 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.656138    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.656445 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.656170    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.656445 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.656254    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9rsh\" (UniqueName: \"kubernetes.io/projected/8e2f4139-a07c-4159-adb4-8fbad0fe1897-kube-api-access-x9rsh\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.656445 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.656274    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.656606 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.656479    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.656606 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.656507    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.656698 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.656614    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.656822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.656796    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.657100 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.657015    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.658740 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.658718    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.658973 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.658958    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.713965 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.713644    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9rsh\" (UniqueName: \"kubernetes.io/projected/8e2f4139-a07c-4159-adb4-8fbad0fe1897-kube-api-access-x9rsh\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.716885 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.716854    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.801143 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.801113    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:41.855216 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:41.854080    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:42.042720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:42.042689    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:42.241969 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:42.241383    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"]
Oct 31 01:19:42.273389 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:19:42.273361    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2f4139_a07c_4159_adb4_8fbad0fe1897.slice/crio-80b1124627265fb974ccf131f2c1ea9672ef5c6d64fe2ff80c3bb9767d2166f6 WatchSource:0}: Error finding container 80b1124627265fb974ccf131f2c1ea9672ef5c6d64fe2ff80c3bb9767d2166f6: Status 404 returned error can't find the container with id 80b1124627265fb974ccf131f2c1ea9672ef5c6d64fe2ff80c3bb9767d2166f6
Oct 31 01:19:42.276138 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:42.276105    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:19:42.276262 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:42.276208    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:19:42.276262 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:42.276241    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:19:42.618703 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:42.618658    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd" event={"ID":"8e2f4139-a07c-4159-adb4-8fbad0fe1897","Type":"ContainerStarted","Data":"89d9c61539e148ce5b133155ea421d08c1d4e529561db6ca71c42caf00ad1cab"}
Oct 31 01:19:42.619216 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:42.618736    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd" event={"ID":"8e2f4139-a07c-4159-adb4-8fbad0fe1897","Type":"ContainerStarted","Data":"80b1124627265fb974ccf131f2c1ea9672ef5c6d64fe2ff80c3bb9767d2166f6"}
Oct 31 01:19:42.742136 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:42.742089    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd" podStartSLOduration=1.742074721 podStartE2EDuration="1.742074721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:42.739307424 +0000 UTC m=+1392.429190551" watchObservedRunningTime="2026-10-31 01:19:42.742074721 +0000 UTC m=+1392.431957848"
Oct 31 01:19:42.801363 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:42.801328    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:43.233423 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:43.233396    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:43.246919 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:43.246891    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:43.635429 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:43.635400    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:19:43.802891 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:43.802854    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:19:43.803064 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:43.802919    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd" podUID="8e2f4139-a07c-4159-adb4-8fbad0fe1897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 01:19:43.904275 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:43.904167    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:44.802037 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:44.802001    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:19:44.802504 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:44.802071    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd" podUID="8e2f4139-a07c-4159-adb4-8fbad0fe1897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 01:19:45.802311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:45.802272    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:19:45.802794 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:45.802331    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd" podUID="8e2f4139-a07c-4159-adb4-8fbad0fe1897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 01:19:46.801898 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:46.801865    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:19:46.802055 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:46.801920    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd" podUID="8e2f4139-a07c-4159-adb4-8fbad0fe1897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 01:19:47.801818 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:47.801782    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:19:47.802241 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:47.801845    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd" podUID="8e2f4139-a07c-4159-adb4-8fbad0fe1897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 01:19:48.707856 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:48.707822    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:48.802462 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:48.802427    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:19:48.802833 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:48.802481    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd" podUID="8e2f4139-a07c-4159-adb4-8fbad0fe1897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 01:19:49.806243 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:49.806213    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:49.806739 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:49.806597    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:49.807227 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:49.807210    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:19:53.485296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:53.485262    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:57.625609 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:57.625565    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:19:57.664339 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:57.664277    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4" podUID="762230e3-0692-4a5b-8e29-13bf8a3a4b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:19:57.752323 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:57.752273    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:19:57.929854 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:57.929747    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:19:58.216371 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:58.216267    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq" podUID="48826212-3a95-460e-9e5e-91ef131c528b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:19:58.456203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:58.434277    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp" podUID="fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:19:58.605256 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:58.561072    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:19:58.682203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:58.670266    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4" podUID="762230e3-0692-4a5b-8e29-13bf8a3a4b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:19:58.682203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:58.670927    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:19:58.757213 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:58.756259    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:19:58.757213 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:58.756944    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:19:58.959367 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:58.959256    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:19:58.991222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:58.959978    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:19:59.232256 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:59.221723    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:59.232256 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:59.221829    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq" podUID="48826212-3a95-460e-9e5e-91ef131c528b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:19:59.232256 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:59.222586    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:19:59.513381 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:59.513260    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp" podUID="fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:19:59.538865 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:59.538837    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:19:59.545349 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:59.545323    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:19:59.548960 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:59.548865    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:19:59.569273 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:59.567262    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 01:19:59.569966 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:59.569879    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:59.590572 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:59.590533    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:19:59.694865 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:59.694838    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:19:59.730522 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:19:59.730486    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:19:59.903645 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:19:59.903616    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:20:00.037367 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:00.037342    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:20:04.326544 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:04.326507    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:05.700860 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:05.700820    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:05.908490 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:05.908459    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:08.599709 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:08.599675    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:09.221064 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:09.221035    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:13.088054 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.088023    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"]
Oct 31 01:20:13.092996 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.092980    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.225881 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.225835    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"]
Oct 31 01:20:13.229992 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.229954    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.230140 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.230112    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc044185-a187-4f47-92ee-972bd7794f22-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.230261 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.230145    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vf94\" (UniqueName: \"kubernetes.io/projected/bc044185-a187-4f47-92ee-972bd7794f22-kube-api-access-5vf94\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.230261 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.230239    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.230386 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.230371    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.230443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.230393    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc044185-a187-4f47-92ee-972bd7794f22-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.230498 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.230478    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.230557 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.230507    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.230557 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.230536    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc044185-a187-4f47-92ee-972bd7794f22-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.331964 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.331934    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.332162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.331977    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc044185-a187-4f47-92ee-972bd7794f22-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.332162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.332038    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.332162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.332073    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.332162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.332117    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc044185-a187-4f47-92ee-972bd7794f22-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.332162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.332162    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.332500 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.332299    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc044185-a187-4f47-92ee-972bd7794f22-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.332500 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.332326    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vf94\" (UniqueName: \"kubernetes.io/projected/bc044185-a187-4f47-92ee-972bd7794f22-kube-api-access-5vf94\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.332500 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.332363    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.332659 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.332532    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.332659 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.332547    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.332886 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.332865    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.333047 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.333029    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.333427 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.333404    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc044185-a187-4f47-92ee-972bd7794f22-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.334665 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.334647    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.334837 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.334818    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc044185-a187-4f47-92ee-972bd7794f22-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.392847 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.392762    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc044185-a187-4f47-92ee-972bd7794f22-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.393603 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.393581    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vf94\" (UniqueName: \"kubernetes.io/projected/bc044185-a187-4f47-92ee-972bd7794f22-kube-api-access-5vf94\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.406628 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.406599    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:13.837242 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.837214    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"]
Oct 31 01:20:13.838381 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:20:13.838356    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc044185_a187_4f47_92ee_972bd7794f22.slice/crio-27bcf987c7198ebc2c4a5b4079513874df6dcfbfe51190c28bc06bb13aa12316 WatchSource:0}: Error finding container 27bcf987c7198ebc2c4a5b4079513874df6dcfbfe51190c28bc06bb13aa12316: Status 404 returned error can't find the container with id 27bcf987c7198ebc2c4a5b4079513874df6dcfbfe51190c28bc06bb13aa12316
Oct 31 01:20:13.840608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.840562    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:20:13.840666 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.840643    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:20:13.840704 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:13.840673    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:20:14.193217 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:14.193112    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:14.294307 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:14.294267    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7" event={"ID":"bc044185-a187-4f47-92ee-972bd7794f22","Type":"ContainerStarted","Data":"41355714b0b6e7f323fc30dcce040dbd85d6c85fd06f0fddae8cf6fe71471418"}
Oct 31 01:20:14.294586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:14.294567    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7" event={"ID":"bc044185-a187-4f47-92ee-972bd7794f22","Type":"ContainerStarted","Data":"27bcf987c7198ebc2c4a5b4079513874df6dcfbfe51190c28bc06bb13aa12316"}
Oct 31 01:20:14.406836 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:14.406800    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:14.408682 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:14.408657    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 01:20:14.408816 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:14.408718    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7" podUID="bc044185-a187-4f47-92ee-972bd7794f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 01:20:14.534905 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:14.534854    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7" podStartSLOduration=1.5348393599999999 podStartE2EDuration="1.53483936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:14.484950207 +0000 UTC m=+1424.174833334" watchObservedRunningTime="2026-10-31 01:20:14.53483936 +0000 UTC m=+1424.224722486"
Oct 31 01:20:15.407599 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:15.407562    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 01:20:15.408090 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:15.407634    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7" podUID="bc044185-a187-4f47-92ee-972bd7794f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 01:20:16.407837 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:16.407806    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 01:20:16.408266 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:16.407870    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7" podUID="bc044185-a187-4f47-92ee-972bd7794f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 01:20:17.407685 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:17.407649    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 01:20:17.407866 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:17.407707    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7" podUID="bc044185-a187-4f47-92ee-972bd7794f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 01:20:18.407958 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:18.407925    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 01:20:18.408396 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:18.407993    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7" podUID="bc044185-a187-4f47-92ee-972bd7794f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 01:20:18.715421 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:18.715338    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:19.167013 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:19.166978    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:19.407358 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:19.407329    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 01:20:19.407524 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:19.407393    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7" podUID="bc044185-a187-4f47-92ee-972bd7794f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 01:20:20.407406 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:20.407369    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 01:20:20.407841 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:20.407439    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7" podUID="bc044185-a187-4f47-92ee-972bd7794f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 01:20:21.411749 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:21.411723    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:21.412249 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:21.412122    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:21.412875 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:21.412854    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:20:23.545612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:23.545543    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4" podUID="762230e3-0692-4a5b-8e29-13bf8a3a4b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:23.561233 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:23.550624    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:23.609200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:23.606868    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:23.633200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:23.629261    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:23.705211 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:23.702271    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:20:23.915403 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:23.915264    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp" podUID="fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:24.029201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.020874    4476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" podUID="27377642-d40d-4498-af5d-16ca6a58ef54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:24.054340 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.054286    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq" podUID="48826212-3a95-460e-9e5e-91ef131c528b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:24.054960 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.054932    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" podUID="27377642-d40d-4498-af5d-16ca6a58ef54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:24.234429 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.234317    4476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:24.571903 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.571860    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4" podUID="762230e3-0692-4a5b-8e29-13bf8a3a4b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:20:24.573639 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.573606    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:24.574898 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.574876    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:20:24.575078 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.575062    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:20:24.616316 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.616271    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:24.616983 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.616958    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:20:24.631575 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.631544    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:20:24.656200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.650128    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:20:24.706202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.705260    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:24.706202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.705694    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:20:24.955520 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.955417    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp" podUID="fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:25.027202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:24.998868    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:20:25.032260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.030268    4476 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq" podUID="48826212-3a95-460e-9e5e-91ef131c528b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:25.159260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.084051    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" podUID="27377642-d40d-4498-af5d-16ca6a58ef54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:25.159260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.084392    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:20:25.159260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.126844    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq" podUID="48826212-3a95-460e-9e5e-91ef131c528b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:25.159260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.127198    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:20:25.326905 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.326853    4476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded"
Oct 31 01:20:25.328017 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.327991    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:20:25.347367 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.346735    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:20:25.635259 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.634726    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:25.677212 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.658292    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:20:25.761274 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.761225    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:25.872217 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.866116    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:25.978612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.978520    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp" podUID="fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:20:25.996685 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.996092    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:20:26.004194 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:25.999770    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:20:26.078106 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:26.078075    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:20:26.188721 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:26.188691    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:20:26.188992 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:26.188976    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:20:26.218436 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:26.218410    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:20:27.442142 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:27.442112    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:29.181862 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:20:29.181829    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:20:31.494445 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:31.494396    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:36.331210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:36.331148    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:41.127049 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:41.126983    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:42.041264 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.041233    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"]
Oct 31 01:20:42.045336 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.045321    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.090712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.090684    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"]
Oct 31 01:20:42.094520 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.094503    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.143334 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.143310    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"]
Oct 31 01:20:42.183464 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.183439    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.183616 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.183476    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlt5d\" (UniqueName: \"kubernetes.io/projected/1c75060f-8290-4054-826e-c0044394f5d5-kube-api-access-tlt5d\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.183616 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.183507    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b499p\" (UniqueName: \"kubernetes.io/projected/b433eeb1-84a1-4010-8e15-070ad662bc27-kube-api-access-b499p\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.183616 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.183600    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.183782 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.183758    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.183821 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.183806    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.183862 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.183848    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c75060f-8290-4054-826e-c0044394f5d5-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.183974 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.183956    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b433eeb1-84a1-4010-8e15-070ad662bc27-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.184035 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.184013    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.184091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.184069    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.184148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.184125    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.184278 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.184261    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.184348 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.184324    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c75060f-8290-4054-826e-c0044394f5d5-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.184404 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.184360    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.184451 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.184411    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.184536 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.184516    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c75060f-8290-4054-826e-c0044394f5d5-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.184576 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.184565    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.184624 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.184610    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.218673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.218640    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"]
Oct 31 01:20:42.285327 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285296    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.285492 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285340    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c75060f-8290-4054-826e-c0044394f5d5-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.285492 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285358    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.285492 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285463    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.285664 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285517    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c75060f-8290-4054-826e-c0044394f5d5-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.285664 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285553    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.285664 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285604    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.285664 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285654    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.285872 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285682    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlt5d\" (UniqueName: \"kubernetes.io/projected/1c75060f-8290-4054-826e-c0044394f5d5-kube-api-access-tlt5d\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.285872 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285714    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b499p\" (UniqueName: \"kubernetes.io/projected/b433eeb1-84a1-4010-8e15-070ad662bc27-kube-api-access-b499p\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.285872 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285739    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.285872 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285820    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.285872 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285852    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.286322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285884    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c75060f-8290-4054-826e-c0044394f5d5-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.286322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285923    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b433eeb1-84a1-4010-8e15-070ad662bc27-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.286322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285954    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.286322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.285992    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.286322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.286008    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.286322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.286015    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.286322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.286086    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.286322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.286239    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.286322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.286276    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.286803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.286415    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.286803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.286483    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c75060f-8290-4054-826e-c0044394f5d5-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.286803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.286653    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.286965 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.286856    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.286965 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.286898    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.287114 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.287096    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b433eeb1-84a1-4010-8e15-070ad662bc27-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.288382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.288356    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.288654 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.288635    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c75060f-8290-4054-826e-c0044394f5d5-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.288817 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.288795    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.289085 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.289064    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.345428 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.345366    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b499p\" (UniqueName: \"kubernetes.io/projected/b433eeb1-84a1-4010-8e15-070ad662bc27-kube-api-access-b499p\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.345544 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.345477    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c75060f-8290-4054-826e-c0044394f5d5-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.345544 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.345535    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlt5d\" (UniqueName: \"kubernetes.io/projected/1c75060f-8290-4054-826e-c0044394f5d5-kube-api-access-tlt5d\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.348378 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.348353    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.358621 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.358604    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:42.407638 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.407607    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:42.791294 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.791269    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"]
Oct 31 01:20:42.791975 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:20:42.791953    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb433eeb1_84a1_4010_8e15_070ad662bc27.slice/crio-df16036aa4494a1650de324e07e425147874a31e3e52fbc908d2587304ada531 WatchSource:0}: Error finding container df16036aa4494a1650de324e07e425147874a31e3e52fbc908d2587304ada531: Status 404 returned error can't find the container with id df16036aa4494a1650de324e07e425147874a31e3e52fbc908d2587304ada531
Oct 31 01:20:42.794356 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.794323    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:20:42.794427 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.794387    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:20:42.794470 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.794436    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:20:42.844864 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.844844    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"]
Oct 31 01:20:42.844992 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:20:42.844968    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c75060f_8290_4054_826e_c0044394f5d5.slice/crio-a58bbd28bbf5b243e166c9a782d28a7e7e4bb1927a0ee9d58a1902f126e42698 WatchSource:0}: Error finding container a58bbd28bbf5b243e166c9a782d28a7e7e4bb1927a0ee9d58a1902f126e42698: Status 404 returned error can't find the container with id a58bbd28bbf5b243e166c9a782d28a7e7e4bb1927a0ee9d58a1902f126e42698
Oct 31 01:20:42.847277 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.847246    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:20:42.847353 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.847309    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:20:42.847353 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:42.847339    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:20:43.013589 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.013558    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" event={"ID":"b433eeb1-84a1-4010-8e15-070ad662bc27","Type":"ContainerStarted","Data":"a2407d28f49213e9b71b18690430cb0b8b1dbab9b6cd3f24a67936ddb150e217"}
Oct 31 01:20:43.013845 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.013825    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" event={"ID":"b433eeb1-84a1-4010-8e15-070ad662bc27","Type":"ContainerStarted","Data":"df16036aa4494a1650de324e07e425147874a31e3e52fbc908d2587304ada531"}
Oct 31 01:20:43.016321 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.016280    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" event={"ID":"1c75060f-8290-4054-826e-c0044394f5d5","Type":"ContainerStarted","Data":"07d7bf5ab9e27ca453cde71f42f3d1d273d3fba0a77ac5df80462aa324adf161"}
Oct 31 01:20:43.016445 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.016346    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" event={"ID":"1c75060f-8290-4054-826e-c0044394f5d5","Type":"ContainerStarted","Data":"a58bbd28bbf5b243e166c9a782d28a7e7e4bb1927a0ee9d58a1902f126e42698"}
Oct 31 01:20:43.131635 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.131527    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" podStartSLOduration=2.131511797 podStartE2EDuration="2.131511797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:43.129947907 +0000 UTC m=+1452.819831035" watchObservedRunningTime="2026-10-31 01:20:43.131511797 +0000 UTC m=+1452.821394924"
Oct 31 01:20:43.247897 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.247837    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" podStartSLOduration=1.247819406 podStartE2EDuration="1.247819406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:43.24743905 +0000 UTC m=+1452.937322179" watchObservedRunningTime="2026-10-31 01:20:43.247819406 +0000 UTC m=+1452.937702534"
Oct 31 01:20:43.319494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.319458    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"]
Oct 31 01:20:43.324744 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.324716    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.359269 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.359236    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:43.377197 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.371288    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:20:43.377197 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.371381    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" podUID="b433eeb1-84a1-4010-8e15-070ad662bc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:20:43.403260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.393961    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"]
Oct 31 01:20:43.409253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.406287    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.409253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.406351    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.409253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.406418    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.409253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.406459    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db07ee58-a36d-485b-aaf1-9f01749469e8-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.409253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.406494    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz68w\" (UniqueName: \"kubernetes.io/projected/db07ee58-a36d-485b-aaf1-9f01749469e8-kube-api-access-fz68w\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.409253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.406541    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.409253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.406574    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.409253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.406600    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.409253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.406791    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.434530 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.433590    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:43.441230 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.439093    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:20:43.441230 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.439149    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" podUID="1c75060f-8290-4054-826e-c0044394f5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:20:43.547016 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.546977    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.547222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.547084    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.547222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.547125    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.547222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.547208    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.547467 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.547247    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db07ee58-a36d-485b-aaf1-9f01749469e8-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.547467 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.547280    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz68w\" (UniqueName: \"kubernetes.io/projected/db07ee58-a36d-485b-aaf1-9f01749469e8-kube-api-access-fz68w\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.547467 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.547328    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.547467 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.547363    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.547467 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.547424    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.554593 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.554559    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.555696 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.555671    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db07ee58-a36d-485b-aaf1-9f01749469e8-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.555936 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.555917    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.556075 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.556049    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.556165 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.556147    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.558926 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.557273    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.559808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.559637    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.613097 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.612992    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz68w\" (UniqueName: \"kubernetes.io/projected/db07ee58-a36d-485b-aaf1-9f01749469e8-kube-api-access-fz68w\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.615335 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.615284    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:43.661248 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:43.660437    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:44.463254 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:44.456235    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:20:44.463254 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:44.456296    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" podUID="1c75060f-8290-4054-826e-c0044394f5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:20:44.463254 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:44.456696    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:20:44.463254 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:44.456728    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" podUID="b433eeb1-84a1-4010-8e15-070ad662bc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:20:44.641567 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:44.641531    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"]
Oct 31 01:20:44.778793 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:44.778737    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:20:44.779087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:44.779060    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:20:44.779272 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:44.779246    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:20:45.007864 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.006267    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" podUID="27377642-d40d-4498-af5d-16ca6a58ef54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:45.028806 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.027671    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:45.047433 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.047334    4476 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq" podUID="48826212-3a95-460e-9e5e-91ef131c528b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:45.140677 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.137946    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq" podUID="48826212-3a95-460e-9e5e-91ef131c528b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:45.230221 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.228115    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:45.230404 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.230240    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:45.283264 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.279815    4476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:45.344212 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.338270    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 01:20:45.370232 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.370199    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:20:45.370518 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.370477    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" podUID="b433eeb1-84a1-4010-8e15-070ad662bc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:20:45.409595 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.409553    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:20:45.409901 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.409858    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" podUID="1c75060f-8290-4054-826e-c0044394f5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:20:45.543767 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.542982    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r" event={"ID":"db07ee58-a36d-485b-aaf1-9f01749469e8","Type":"ContainerStarted","Data":"68f27dbd91878b9748d212ae74c2fa49d582135c3ec1721b96854cec1cb8ca45"}
Oct 31 01:20:45.543767 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.543043    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r" event={"ID":"db07ee58-a36d-485b-aaf1-9f01749469e8","Type":"ContainerStarted","Data":"007979ee3dcf8d26fb36dd96dc56c4ae07b229b44aa51d6692c15d43cf4835c6"}
Oct 31 01:20:45.662292 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:45.662259    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:46.022897 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.022791    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" podUID="27377642-d40d-4498-af5d-16ca6a58ef54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:46.024096 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.024074    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:20:46.040593 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.040552    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:46.041230 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.041209    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:20:46.351199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.315537    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:46.376315 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.351716    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq" podUID="48826212-3a95-460e-9e5e-91ef131c528b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:46.399204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.387493    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 31 01:20:46.490235 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.470703    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:20:46.490235 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.470914    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:20:46.490235 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.471108    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:20:46.508562 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.493322    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:20:46.508562 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.493388    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" podUID="b433eeb1-84a1-4010-8e15-070ad662bc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:20:46.799275 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.773827    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 01:20:46.807363 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.776777    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4" podUID="762230e3-0692-4a5b-8e29-13bf8a3a4b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:20:46.807628 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.776926    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp" podUID="fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:47.002203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:46.977025    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r" podStartSLOduration=3.977002975 podStartE2EDuration="3.977002975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:45.693132027 +0000 UTC m=+1455.383015152" watchObservedRunningTime="2026-10-31 01:20:46.977002975 +0000 UTC m=+1456.666886104"
Oct 31 01:20:47.062271 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.036435    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" podUID="27377642-d40d-4498-af5d-16ca6a58ef54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:47.456862 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.105006    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:20:47.457153 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.457100    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r" podUID="db07ee58-a36d-485b-aaf1-9f01749469e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:20:47.457326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.316804    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:47.457796 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.231164    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:47.458199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.182857    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:20:47.461319 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.461280    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" podUID="1c75060f-8290-4054-826e-c0044394f5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:20:47.530809 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.396647    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:47.577203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.388722    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:20:47.577203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.558624    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" podUID="b433eeb1-84a1-4010-8e15-070ad662bc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:20:47.606203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.587256    4476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:47.630203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.618687    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:47.649202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.649152    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:20:47.657302 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.657275    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:20:47.681830 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.681794    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:20:47.682106 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.682076    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r" podUID="db07ee58-a36d-485b-aaf1-9f01749469e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:20:47.801254 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.755413    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:20:47.801254 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.755474    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" podUID="1c75060f-8290-4054-826e-c0044394f5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:20:47.846219 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.829709    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 01:20:47.846219 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.829785    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 01:20:47.852204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.849118    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:48.024236 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:47.998573    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Oct 31 01:20:48.073667 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:48.073568    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4" podUID="762230e3-0692-4a5b-8e29-13bf8a3a4b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:48.074219 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:48.074199    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:20:48.354452 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:48.354378    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:20:48.389210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:48.385590    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:20:48.389210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:48.385648    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" podUID="b433eeb1-84a1-4010-8e15-070ad662bc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:20:48.409628 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:48.409595    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:20:48.439451 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:48.439414    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" podUID="1c75060f-8290-4054-826e-c0044394f5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:20:48.474302 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:48.474253    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:20:48.651606 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:48.651509    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:20:48.651606 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:48.651602    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 01:20:48.651850 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:48.651651    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 01:20:48.703200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:48.696562    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:20:48.703200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:48.696645    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r" podUID="db07ee58-a36d-485b-aaf1-9f01749469e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:20:49.408203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:49.407312    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:20:49.408203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:49.407475    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" podUID="b433eeb1-84a1-4010-8e15-070ad662bc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:20:49.435398 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:49.408434    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:20:49.435398 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:49.408476    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" podUID="1c75060f-8290-4054-826e-c0044394f5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:20:49.669770 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:49.669668    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:20:49.670079 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:49.670029    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r" podUID="db07ee58-a36d-485b-aaf1-9f01749469e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:20:50.363589 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:50.363551    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:20:50.363885 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:50.363846    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" podUID="b433eeb1-84a1-4010-8e15-070ad662bc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:20:50.408552 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:50.408521    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:20:50.409199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:50.409146    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" podUID="1c75060f-8290-4054-826e-c0044394f5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:20:50.663564 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:50.663474    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:20:50.668291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:50.668248    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r" podUID="db07ee58-a36d-485b-aaf1-9f01749469e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:20:51.364201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:51.363534    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:20:51.364201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:51.363594    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" podUID="b433eeb1-84a1-4010-8e15-070ad662bc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:20:51.407984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:51.407947    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:20:51.408188 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:51.408016    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" podUID="1c75060f-8290-4054-826e-c0044394f5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:20:51.668047 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:51.663596    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:20:51.668047 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:51.663658    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r" podUID="db07ee58-a36d-485b-aaf1-9f01749469e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:20:52.373202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:52.369454    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:52.373202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:52.370805    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:52.380197 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:52.375369    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:20:52.662198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:52.662091    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:20:52.662355 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:52.662195    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r" podUID="db07ee58-a36d-485b-aaf1-9f01749469e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 01:20:52.743639 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:52.743603    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:52.744424 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:52.744401    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:52.745368 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:52.745350    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:20:53.381201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:53.381145    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:53.949421 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:53.949378    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:53.950353 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:53.950324    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:53.951838 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:53.951820    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:20:58.444507 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:58.444470    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:59.203411 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:20:59.203369    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:20:59.419491 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:59.419458    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"]
Oct 31 01:20:59.419725 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:20:59.419702    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph" podUID="c0eace8b-08cd-43a8-b4f9-c5eacce5135e" containerName="istio-proxy" containerID="cri-o://673a311a3d58e21073793faba4b4d0c7b6ea28762506c1bbd520f4468b503b03" gracePeriod=30
Oct 31 01:21:01.264201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:01.260273    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq" podUID="48826212-3a95-460e-9e5e-91ef131c528b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:01.450240 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:01.450199    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" podUID="27377642-d40d-4498-af5d-16ca6a58ef54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 01:21:01.475287 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:01.475246    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:01.476029 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:01.476002    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:01.832199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:01.831264    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:01.832199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:01.831673    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:21:03.148907 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:03.148873    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:03.799891 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:03.799854    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:21:03.800059 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:03.799936    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph" podUID="c0eace8b-08cd-43a8-b4f9-c5eacce5135e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:21:03.918451 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:03.918413    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:04.323055 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:04.323024    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:05.990041 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:05.990017    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:21:06.096883 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.096853    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-kube-api-access-np47s\" (UniqueName: \"kubernetes.io/projected/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-kube-api-access-np47s\") pod \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") "
Oct 31 01:21:06.097035 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.096901    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-envoy\") pod \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") "
Oct 31 01:21:06.097035 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.097005    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-workload-socket\") pod \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") "
Oct 31 01:21:06.097158 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.097085    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-podinfo\") pod \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") "
Oct 31 01:21:06.097158 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.097122    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-workload-certs\") pod \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") "
Oct 31 01:21:06.097282 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.097233    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-credential-socket\") pod \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") "
Oct 31 01:21:06.097282 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.097266    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-token\" (UniqueName: \"kubernetes.io/projected/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-token\") pod \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") "
Oct 31 01:21:06.097391 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.097308    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istiod-ca-cert\") pod \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") "
Oct 31 01:21:06.097391 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.097326    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-workload-socket" pod "c0eace8b-08cd-43a8-b4f9-c5eacce5135e" (UID: "c0eace8b-08cd-43a8-b4f9-c5eacce5135e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:21:06.097514 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.097408    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-data\") pod \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\" (UID: \"c0eace8b-08cd-43a8-b4f9-c5eacce5135e\") "
Oct 31 01:21:06.097601 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.097576    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-workload-certs" pod "c0eace8b-08cd-43a8-b4f9-c5eacce5135e" (UID: "c0eace8b-08cd-43a8-b4f9-c5eacce5135e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:21:06.097732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.097609    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-credential-socket" pod "c0eace8b-08cd-43a8-b4f9-c5eacce5135e" (UID: "c0eace8b-08cd-43a8-b4f9-c5eacce5135e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:21:06.097806 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.097744    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-data" pod "c0eace8b-08cd-43a8-b4f9-c5eacce5135e" (UID: "c0eace8b-08cd-43a8-b4f9-c5eacce5135e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:21:06.097806 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.097784    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istiod-ca-cert" pod "c0eace8b-08cd-43a8-b4f9-c5eacce5135e" (UID: "c0eace8b-08cd-43a8-b4f9-c5eacce5135e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:21:06.098293 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.098271    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:21:06.098402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.098300    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:21:06.098402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.098316    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:21:06.098402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.098331    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:21:06.098402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.098344    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:21:06.099601 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.099575    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-kube-api-access-np47s" pod "c0eace8b-08cd-43a8-b4f9-c5eacce5135e" (UID: "c0eace8b-08cd-43a8-b4f9-c5eacce5135e"). InnerVolumeSpecName "kube-api-access-np47s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:21:06.099601 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.099591    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-podinfo" pod "c0eace8b-08cd-43a8-b4f9-c5eacce5135e" (UID: "c0eace8b-08cd-43a8-b4f9-c5eacce5135e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:21:06.100020 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.100001    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-envoy" pod "c0eace8b-08cd-43a8-b4f9-c5eacce5135e" (UID: "c0eace8b-08cd-43a8-b4f9-c5eacce5135e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:21:06.100087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.100031    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-token" pod "c0eace8b-08cd-43a8-b4f9-c5eacce5135e" (UID: "c0eace8b-08cd-43a8-b4f9-c5eacce5135e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:21:06.199519 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.199484    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np47s\" (UniqueName: \"kubernetes.io/projected/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-kube-api-access-np47s\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:21:06.199519 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.199521    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:21:06.199711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.199537    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:21:06.199711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.199550    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0eace8b-08cd-43a8-b4f9-c5eacce5135e-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:21:06.457769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.457665    4476 generic.go:356] "Generic (PLEG): container finished" podID="c0eace8b-08cd-43a8-b4f9-c5eacce5135e" containerID="673a311a3d58e21073793faba4b4d0c7b6ea28762506c1bbd520f4468b503b03" exitCode=0
Oct 31 01:21:06.457769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.457726    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"
Oct 31 01:21:06.457769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.457742    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph" event={"ID":"c0eace8b-08cd-43a8-b4f9-c5eacce5135e","Type":"ContainerDied","Data":"673a311a3d58e21073793faba4b4d0c7b6ea28762506c1bbd520f4468b503b03"}
Oct 31 01:21:06.458038 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.457807    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph" event={"ID":"c0eace8b-08cd-43a8-b4f9-c5eacce5135e","Type":"ContainerDied","Data":"043cfbf93af79af8e419bebd3df8a28f17e10fe5971fc0062cb4d27819a3c35e"}
Oct 31 01:21:06.458038 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.457836    4476 scope.go:122] "RemoveContainer" containerID="673a311a3d58e21073793faba4b4d0c7b6ea28762506c1bbd520f4468b503b03"
Oct 31 01:21:06.475912 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.475897    4476 scope.go:122] "RemoveContainer" containerID="673a311a3d58e21073793faba4b4d0c7b6ea28762506c1bbd520f4468b503b03"
Oct 31 01:21:06.476198 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:21:06.476165    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"673a311a3d58e21073793faba4b4d0c7b6ea28762506c1bbd520f4468b503b03\": container with ID starting with 673a311a3d58e21073793faba4b4d0c7b6ea28762506c1bbd520f4468b503b03 not found: ID does not exist" containerID="673a311a3d58e21073793faba4b4d0c7b6ea28762506c1bbd520f4468b503b03"
Oct 31 01:21:06.476252 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.476207    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"673a311a3d58e21073793faba4b4d0c7b6ea28762506c1bbd520f4468b503b03"} err="failed to get container status \"673a311a3d58e21073793faba4b4d0c7b6ea28762506c1bbd520f4468b503b03\": rpc error: code = NotFound desc = could not find container \"673a311a3d58e21073793faba4b4d0c7b6ea28762506c1bbd520f4468b503b03\": container with ID starting with 673a311a3d58e21073793faba4b4d0c7b6ea28762506c1bbd520f4468b503b03 not found: ID does not exist"
Oct 31 01:21:06.627078 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.627044    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"]
Oct 31 01:21:06.676340 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:06.676315    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pxxph"]
Oct 31 01:21:08.320410 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:08.320383    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:08.456494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:08.456461    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0eace8b-08cd-43a8-b4f9-c5eacce5135e" path="/var/lib/kubelet/pods/c0eace8b-08cd-43a8-b4f9-c5eacce5135e/volumes"
Oct 31 01:21:08.607716 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:08.607632    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:08.893485 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:08.893414    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:09.266163 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.266055    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:09.701569 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.701531    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"]
Oct 31 01:21:09.703279 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.703251    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c0eace8b-08cd-43a8-b4f9-c5eacce5135e" containerName="istio-proxy"
Oct 31 01:21:09.703446 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.703431    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0eace8b-08cd-43a8-b4f9-c5eacce5135e" containerName="istio-proxy"
Oct 31 01:21:09.703881 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.703862    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0eace8b-08cd-43a8-b4f9-c5eacce5135e" containerName="istio-proxy"
Oct 31 01:21:09.709363 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.709343    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.739272 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.739242    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-dshm\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.739422 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.739296    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-tls-certs\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.739532 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.739503    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-model-cache\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.739598 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.739549    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-tmp-dir\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.739656 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.739628    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg97w\" (UniqueName: \"kubernetes.io/projected/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-kube-api-access-cg97w\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.739728 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.739702    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-home\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.769047 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.769016    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"]
Oct 31 01:21:09.814520 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.814490    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ec695b\"/\"distinct-2-w3-ec695b-kserve-self-signed-certs\""
Oct 31 01:21:09.841527 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.841464    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-dshm\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.841742 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.841532    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-tls-certs\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.841742 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.841650    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-model-cache\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.841742 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.841679    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-tmp-dir\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.841742 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.841728    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg97w\" (UniqueName: \"kubernetes.io/projected/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-kube-api-access-cg97w\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.842127 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.841774    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-home\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.842432 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.842304    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-model-cache\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.842432 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.842381    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-home\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.843406 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.842946    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-tmp-dir\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.851880 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.851819    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-tls-certs\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.860588 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.860531    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-dshm\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:09.862880 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.862771    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:09.899510 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:09.899482    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg97w\" (UniqueName: \"kubernetes.io/projected/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-kube-api-access-cg97w\") pod \"distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") " pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:10.022214 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:10.022108    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:10.431730 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:10.431700    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"]
Oct 31 01:21:10.432891 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:21:10.432859    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dcbe11f_bf2c_4e64_91db_96868f1f38ea.slice/crio-34518805170b3aeb2d23cf5857e08d6b3c6e7660ac5de972829c9c5d6170c679 WatchSource:0}: Error finding container 34518805170b3aeb2d23cf5857e08d6b3c6e7660ac5de972829c9c5d6170c679: Status 404 returned error can't find the container with id 34518805170b3aeb2d23cf5857e08d6b3c6e7660ac5de972829c9c5d6170c679
Oct 31 01:21:10.495826 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:10.495801    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv" event={"ID":"9dcbe11f-bf2c-4e64-91db-96868f1f38ea","Type":"ContainerStarted","Data":"34518805170b3aeb2d23cf5857e08d6b3c6e7660ac5de972829c9c5d6170c679"}
Oct 31 01:21:11.503222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:11.503194    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv" event={"ID":"9dcbe11f-bf2c-4e64-91db-96868f1f38ea","Type":"ContainerStarted","Data":"e7db36c19eebde364216ad894264f07e6286f0ff6506c2a35785b3d4fea73948"}
Oct 31 01:21:11.623828 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:11.623769    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv" podStartSLOduration=2.62375062 podStartE2EDuration="2.62375062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:11.620418776 +0000 UTC m=+1481.310301982" watchObservedRunningTime="2026-10-31 01:21:11.62375062 +0000 UTC m=+1481.313633747"
Oct 31 01:21:11.757457 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:11.757373    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:11.985011 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:11.984970    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:12.023069 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:12.022980    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:14.542985 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:14.542944    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:18.280021 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:18.279991    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:18.551902 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:18.551806    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:18.998247 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:18.998113    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:19.432044 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:19.432008    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:19.788639 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:19.788605    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:20.023009 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:20.022978    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:20.035783 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:20.035761    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:20.356941 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:20.356912    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:20.569347 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:20.569319    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:24.630191 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:24.630144    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:26.198239 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:26.198193    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp" podUID="fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:26.276337 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:26.276286    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq" podUID="48826212-3a95-460e-9e5e-91ef131c528b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:26.457260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:26.455315    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4" podUID="762230e3-0692-4a5b-8e29-13bf8a3a4b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:26.457260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:26.456510    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:26.457260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:26.456577    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" podUID="27377642-d40d-4498-af5d-16ca6a58ef54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 01:21:26.461304 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:26.461250    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:26.569321 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:26.569268    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv" podUID="9dcbe11f-bf2c-4e64-91db-96868f1f38ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:26.869268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:26.846282    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:26.869268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:26.847865    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:27.199356 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.199240    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp" podUID="fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 01:21:27.200545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.200522    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:21:27.288690 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.288639    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq" podUID="48826212-3a95-460e-9e5e-91ef131c528b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:27.291058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.291025    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:21:27.495531 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.495393    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:27.508223 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.507003    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" podUID="27377642-d40d-4498-af5d-16ca6a58ef54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 01:21:27.508223 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.508131    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:21:27.509058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.508996    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:21:27.529301 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.529264    4476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:27.578133 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.578089    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv" podUID="9dcbe11f-bf2c-4e64-91db-96868f1f38ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:21:27.578820 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.578728    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:27.623883 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.623858    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 01:21:27.729318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.729293    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 01:21:27.731014 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.730995    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq"
Oct 31 01:21:27.738205 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.737414    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 01:21:27.739093 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:27.739071    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 01:21:29.182267 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:21:29.182230    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:21:29.938160 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:29.938123    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:30.617740 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:30.617708    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:21:30.635137 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:30.635111    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:21:34.690913 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:34.690878    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:37.852055 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:37.852022    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:39.190318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.190275    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"]
Oct 31 01:21:39.195388 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.195371    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:39.247501 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.247475    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f259852e-serving-cert\""
Oct 31 01:21:39.290356 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.290322    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-maas-api-tls\") pod \"maas-api-e2e-mt-f259852e-56d9866d6-qsqqd\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:39.290502 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.290485    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-metrics-certs\") pod \"maas-api-e2e-mt-f259852e-56d9866d6-qsqqd\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:39.290565 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.290549    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f84534-7aab-4133-8f6c-a6b2fa4983fb-openshift-service-ca\") pod \"maas-api-e2e-mt-f259852e-56d9866d6-qsqqd\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:39.290610 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.290578    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxswt\" (UniqueName: \"kubernetes.io/projected/46f84534-7aab-4133-8f6c-a6b2fa4983fb-kube-api-access-kxswt\") pod \"maas-api-e2e-mt-f259852e-56d9866d6-qsqqd\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:39.300251 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.300228    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"]
Oct 31 01:21:39.392305 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.392273    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-metrics-certs\") pod \"maas-api-e2e-mt-f259852e-56d9866d6-qsqqd\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:39.392467 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.392349    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f84534-7aab-4133-8f6c-a6b2fa4983fb-openshift-service-ca\") pod \"maas-api-e2e-mt-f259852e-56d9866d6-qsqqd\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:39.392467 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.392369    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxswt\" (UniqueName: \"kubernetes.io/projected/46f84534-7aab-4133-8f6c-a6b2fa4983fb-kube-api-access-kxswt\") pod \"maas-api-e2e-mt-f259852e-56d9866d6-qsqqd\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:39.392561 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.392469    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-maas-api-tls\") pod \"maas-api-e2e-mt-f259852e-56d9866d6-qsqqd\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:39.392604 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:21:39.392589    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-serving-cert: secret "maas-api-e2e-mt-f259852e-serving-cert" not found
Oct 31 01:21:39.392665 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:21:39.392654    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-maas-api-tls podName:46f84534-7aab-4133-8f6c-a6b2fa4983fb nodeName:}" failed. No retries permitted until 2026-10-31 01:21:39.89263553 +0000 UTC m=+1509.582518638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-maas-api-tls") pod "maas-api-e2e-mt-f259852e-56d9866d6-qsqqd" (UID: "46f84534-7aab-4133-8f6c-a6b2fa4983fb") : secret "maas-api-e2e-mt-f259852e-serving-cert" not found
Oct 31 01:21:39.393053 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.393026    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f84534-7aab-4133-8f6c-a6b2fa4983fb-openshift-service-ca\") pod \"maas-api-e2e-mt-f259852e-56d9866d6-qsqqd\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:39.394962 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.394942    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-metrics-certs\") pod \"maas-api-e2e-mt-f259852e-56d9866d6-qsqqd\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:39.429871 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.429844    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:39.458678 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.458315    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxswt\" (UniqueName: \"kubernetes.io/projected/46f84534-7aab-4133-8f6c-a6b2fa4983fb-kube-api-access-kxswt\") pod \"maas-api-e2e-mt-f259852e-56d9866d6-qsqqd\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:39.607655 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.607622    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:39.782473 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.782440    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:39.900083 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.900052    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-maas-api-tls\") pod \"maas-api-e2e-mt-f259852e-56d9866d6-qsqqd\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:39.903514 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:39.903486    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-maas-api-tls\") pod \"maas-api-e2e-mt-f259852e-56d9866d6-qsqqd\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:40.108008 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:40.107933    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:40.482137 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:40.482110    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"]
Oct 31 01:21:40.483003 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:21:40.482979    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f84534_7aab_4133_8f6c_a6b2fa4983fb.slice/crio-3d2dd3909c8afecc48b115217499b8273181e26326dbdc77aa1aa6debbd66be6 WatchSource:0}: Error finding container 3d2dd3909c8afecc48b115217499b8273181e26326dbdc77aa1aa6debbd66be6: Status 404 returned error can't find the container with id 3d2dd3909c8afecc48b115217499b8273181e26326dbdc77aa1aa6debbd66be6
Oct 31 01:21:40.484242 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:40.484222    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:21:40.824092 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:40.824062    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd" event={"ID":"46f84534-7aab-4133-8f6c-a6b2fa4983fb","Type":"ContainerStarted","Data":"3d2dd3909c8afecc48b115217499b8273181e26326dbdc77aa1aa6debbd66be6"}
Oct 31 01:21:41.831907 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:41.831871    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd" event={"ID":"46f84534-7aab-4133-8f6c-a6b2fa4983fb","Type":"ContainerStarted","Data":"109020bf8ccfe7ec3b9fc309b73cc4502ec7f0133fa442b061e36bcefe256a4d"}
Oct 31 01:21:41.832358 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:41.831947    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:41.953721 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:41.953673    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd" podStartSLOduration=2.001680709 podStartE2EDuration="2.953660535s" totalImagesPullingTime="951.979826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:41.953313387 +0000 UTC m=+1511.643196518" watchObservedRunningTime="2026-10-31 01:21:41.953660535 +0000 UTC m=+1511.643543662"
Oct 31 01:21:44.363911 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:44.363884    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:44.549295 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:44.549259    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:46.372253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:46.372216    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:47.854692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:47.847873    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:21:49.444127 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:49.444074    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:54.306796 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:54.306751    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:59.020847 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:21:59.020816    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:59.182068 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:21:59.182024    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:22:03.838410 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:03.838372    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:08.494363 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:08.494324    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:08.675671 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:08.675641    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:09.287890 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:09.287853    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:13.193087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:13.193050    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:17.989904 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:17.989869    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:19.282484 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:19.282453    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:19.615086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:19.614996    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:23.125155 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:23.125124    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:26.579054 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:26.579019    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:27.238477 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:27.238444    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:27.489395 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:27.489315    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:27.884120 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:27.884087    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:29.185093 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:22:29.185049    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:22:31.807755 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:31.807722    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:32.002441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:32.002371    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:32.840372 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:32.840297    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:34.994387 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:34.994358    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:35.167624 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:35.167586    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:35.522422 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:35.522392    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"]
Oct 31 01:22:35.527268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:35.527240    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"
Oct 31 01:22:35.612267 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:35.600234    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rchs\" (UniqueName: \"kubernetes.io/projected/640ee4f4-6d6d-49a8-b2b1-686e39623b85-kube-api-access-9rchs\") pod \"payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg\" (UID: \"640ee4f4-6d6d-49a8-b2b1-686e39623b85\") " pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"
Oct 31 01:22:35.612267 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:35.600377    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640ee4f4-6d6d-49a8-b2b1-686e39623b85-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg\" (UID: \"640ee4f4-6d6d-49a8-b2b1-686e39623b85\") " pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"
Oct 31 01:22:35.612267 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:35.600762    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"]
Oct 31 01:22:35.648525 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:35.646506    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4bcda6-dockercfg-ljl8j\""
Oct 31 01:22:35.648525 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:35.647156    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4bcda6\""
Oct 31 01:22:35.701449 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:35.701417    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rchs\" (UniqueName: \"kubernetes.io/projected/640ee4f4-6d6d-49a8-b2b1-686e39623b85-kube-api-access-9rchs\") pod \"payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg\" (UID: \"640ee4f4-6d6d-49a8-b2b1-686e39623b85\") " pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"
Oct 31 01:22:35.701664 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:35.701570    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640ee4f4-6d6d-49a8-b2b1-686e39623b85-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg\" (UID: \"640ee4f4-6d6d-49a8-b2b1-686e39623b85\") " pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"
Oct 31 01:22:35.702519 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:35.702389    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640ee4f4-6d6d-49a8-b2b1-686e39623b85-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg\" (UID: \"640ee4f4-6d6d-49a8-b2b1-686e39623b85\") " pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"
Oct 31 01:22:35.760028 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:35.759999    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rchs\" (UniqueName: \"kubernetes.io/projected/640ee4f4-6d6d-49a8-b2b1-686e39623b85-kube-api-access-9rchs\") pod \"payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg\" (UID: \"640ee4f4-6d6d-49a8-b2b1-686e39623b85\") " pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"
Oct 31 01:22:35.779087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:35.779058    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:35.841294 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:35.841265    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"
Oct 31 01:22:36.219831 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:36.219790    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"]
Oct 31 01:22:36.222905 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:22:36.222877    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640ee4f4_6d6d_49a8_b2b1_686e39623b85.slice/crio-448c319129dfc713a7269b0c8a954a6160200c72be7ce4aaba4ef1b4b91115c0 WatchSource:0}: Error finding container 448c319129dfc713a7269b0c8a954a6160200c72be7ce4aaba4ef1b4b91115c0: Status 404 returned error can't find the container with id 448c319129dfc713a7269b0c8a954a6160200c72be7ce4aaba4ef1b4b91115c0
Oct 31 01:22:36.238824 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:36.238797    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg" event={"ID":"640ee4f4-6d6d-49a8-b2b1-686e39623b85","Type":"ContainerStarted","Data":"448c319129dfc713a7269b0c8a954a6160200c72be7ce4aaba4ef1b4b91115c0"}
Oct 31 01:22:37.248475 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:37.248429    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg" event={"ID":"640ee4f4-6d6d-49a8-b2b1-686e39623b85","Type":"ContainerStarted","Data":"747b54820c9ce8a77bc870e76de85127cb12c55d399fc2b77b4d2c92bb60b830"}
Oct 31 01:22:37.249316 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:37.249289    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"
Oct 31 01:22:37.363201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:37.363136    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg" podStartSLOduration=2.363121288 podStartE2EDuration="2.363121288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:37.361842523 +0000 UTC m=+1567.051725650" watchObservedRunningTime="2026-10-31 01:22:37.363121288 +0000 UTC m=+1567.053004415"
Oct 31 01:22:37.796089 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:37.796049    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:42.468824 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:42.468739    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:45.278199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.276291    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"]
Oct 31 01:22:45.298479 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.298446    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.352704 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.352674    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"]
Oct 31 01:22:45.398151 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.398112    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-26fbc4-openshift-default-dockercfg-bscl8\""
Oct 31 01:22:45.445108 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.445035    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.445108 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.445098    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.445416 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.445136    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65056fe-146a-4d91-ac90-964f71f943a7-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.445416 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.445308    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.445568 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.445416    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65056fe-146a-4d91-ac90-964f71f943a7-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.445568 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.445480    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmrlk\" (UniqueName: \"kubernetes.io/projected/e65056fe-146a-4d91-ac90-964f71f943a7-kube-api-access-tmrlk\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.445692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.445619    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.445692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.445641    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.445762 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.445730    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65056fe-146a-4d91-ac90-964f71f943a7-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.547096 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.547017    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65056fe-146a-4d91-ac90-964f71f943a7-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.547096 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.547085    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmrlk\" (UniqueName: \"kubernetes.io/projected/e65056fe-146a-4d91-ac90-964f71f943a7-kube-api-access-tmrlk\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.547371 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.547270    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.547371 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.547305    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.547485 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.547383    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65056fe-146a-4d91-ac90-964f71f943a7-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.547567 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.547546    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.547671 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.547654    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.547741 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.547652    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.547796 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.547742    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65056fe-146a-4d91-ac90-964f71f943a7-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.547796 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.547777    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.547912 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.547834    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65056fe-146a-4d91-ac90-964f71f943a7-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.547971 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.547952    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.548017 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.548002    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.548053 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.548023    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.550189 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.550148    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.550480 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.550462    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65056fe-146a-4d91-ac90-964f71f943a7-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.604939 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.604916    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65056fe-146a-4d91-ac90-964f71f943a7-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.605127 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.605108    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmrlk\" (UniqueName: \"kubernetes.io/projected/e65056fe-146a-4d91-ac90-964f71f943a7-kube-api-access-tmrlk\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:45.617462 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:45.617442    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:46.045441 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:22:46.045414    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65056fe_146a_4d91_ac90_964f71f943a7.slice/crio-1e242313339913103bc2f6fd90eef409f2f6eb10eced0f6472b674b17d307f07 WatchSource:0}: Error finding container 1e242313339913103bc2f6fd90eef409f2f6eb10eced0f6472b674b17d307f07: Status 404 returned error can't find the container with id 1e242313339913103bc2f6fd90eef409f2f6eb10eced0f6472b674b17d307f07
Oct 31 01:22:46.045704 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:46.045678    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"]
Oct 31 01:22:46.048012 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:46.047979    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:22:46.048116 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:46.048044    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:22:46.048116 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:46.048070    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:22:46.314725 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:46.314625    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m" event={"ID":"e65056fe-146a-4d91-ac90-964f71f943a7","Type":"ContainerStarted","Data":"3fee524746cf85c00dd014cf730d607010e25f2f848dcfb099b8e7dcc889ed17"}
Oct 31 01:22:46.314725 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:46.314685    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m" event={"ID":"e65056fe-146a-4d91-ac90-964f71f943a7","Type":"ContainerStarted","Data":"1e242313339913103bc2f6fd90eef409f2f6eb10eced0f6472b674b17d307f07"}
Oct 31 01:22:46.432230 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:46.432156    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m" podStartSLOduration=1.4321400579999999 podStartE2EDuration="1.432140058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:46.430360817 +0000 UTC m=+1576.120243944" watchObservedRunningTime="2026-10-31 01:22:46.432140058 +0000 UTC m=+1576.122023183"
Oct 31 01:22:46.617877 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:46.617792    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:47.231611 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:47.231574    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:47.619922 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:47.619890    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:22:47.620324 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:47.619941    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m" podUID="e65056fe-146a-4d91-ac90-964f71f943a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:22:48.618211 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:48.618131    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:22:48.618421 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:48.618220    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m" podUID="e65056fe-146a-4d91-ac90-964f71f943a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:22:49.284596 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:49.284568    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"
Oct 31 01:22:49.618658 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:49.618566    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:22:49.618658 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:49.618621    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m" podUID="e65056fe-146a-4d91-ac90-964f71f943a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:22:50.617854 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:50.617814    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:22:50.618284 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:50.617879    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m" podUID="e65056fe-146a-4d91-ac90-964f71f943a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:22:51.618674 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:51.618627    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:22:51.619088 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:51.618698    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m" podUID="e65056fe-146a-4d91-ac90-964f71f943a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:22:52.025958 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:52.025858    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:52.618487 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:52.618451    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:22:52.618705 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:52.618511    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m" podUID="e65056fe-146a-4d91-ac90-964f71f943a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:22:53.622626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:53.622596    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:53.623112 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:53.623010    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:53.623757 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:53.623737    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:22:55.815028 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:55.814996    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"]
Oct 31 01:22:55.818882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:55.818868    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:55.870723 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:55.870694    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 01:22:55.870861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:55.870740    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 01:22:55.875199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:55.871274    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 01:22:55.875199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:55.871373    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dtmj7\""
Oct 31 01:22:55.904613 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:55.904586    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:55.904733 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:55.904622    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmxst\" (UniqueName: \"kubernetes.io/projected/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-kube-api-access-nmxst\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:55.904733 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:55.904723    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:55.904802 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:55.904780    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:55.904888 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:55.904868    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:55.904992 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:55.904980    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:55.905029 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:55.905001    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:55.924233 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:55.924206    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"]
Oct 31 01:22:56.006523 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.006492    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.006708 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.006560    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.006708 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.006590    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.006708 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.006674    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.006708 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.006689    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.006929 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.006741    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.006929 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.006760    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmxst\" (UniqueName: \"kubernetes.io/projected/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-kube-api-access-nmxst\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.007491 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.007437    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.009037 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.009018    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.009133 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.009060    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.009575 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.009536    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.009575 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.009567    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.063545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.063515    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmxst\" (UniqueName: \"kubernetes.io/projected/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-kube-api-access-nmxst\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.063545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.063541    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhmdw\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.131965 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.131901    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:56.567374 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.567349    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"]
Oct 31 01:22:56.568238 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:22:56.568211    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff675b78_d5fa_4020_90d5_4ed8fa5162ce.slice/crio-989449e7d38606f3e3dd18876b2738c5497ebe01b33d0a924840726cdd496330 WatchSource:0}: Error finding container 989449e7d38606f3e3dd18876b2738c5497ebe01b33d0a924840726cdd496330: Status 404 returned error can't find the container with id 989449e7d38606f3e3dd18876b2738c5497ebe01b33d0a924840726cdd496330
Oct 31 01:22:56.570485 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.570450    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:22:56.570555 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.570513    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:22:56.865937 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:56.865859    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:57.380367 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:57.380331    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw" event={"ID":"ff675b78-d5fa-4020-90d5-4ed8fa5162ce","Type":"ContainerStarted","Data":"4b6be4cab381db9ca73ef79a0814687e76bd6a750bfcf81dca3b62f9eb3a0b60"}
Oct 31 01:22:57.380561 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:57.380389    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:22:57.380561 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:57.380409    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw" event={"ID":"ff675b78-d5fa-4020-90d5-4ed8fa5162ce","Type":"ContainerStarted","Data":"989449e7d38606f3e3dd18876b2738c5497ebe01b33d0a924840726cdd496330"}
Oct 31 01:22:57.382240 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:57.382212    4476 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vhmdw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:22:57.382348 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:57.382262    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw" podUID="ff675b78-d5fa-4020-90d5-4ed8fa5162ce" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:22:57.586312 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:57.586261    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw" podStartSLOduration=2.586248318 podStartE2EDuration="2.586248318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:57.585080909 +0000 UTC m=+1587.274964038" watchObservedRunningTime="2026-10-31 01:22:57.586248318 +0000 UTC m=+1587.276131444"
Oct 31 01:22:58.386673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:58.386644    4476 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vhmdw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:22:58.387087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:22:58.386703    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw" podUID="ff675b78-d5fa-4020-90d5-4ed8fa5162ce" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:22:59.181585 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:22:59.181547    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:23:01.387097 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:01.387069    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:23:01.587796 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:01.587739    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:06.237451 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:06.237410    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:06.415560 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:06.415524    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:08.975652 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:08.975618    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:11.179551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:11.179520    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:12.095009 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:12.094977    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:12.270990 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:12.270961    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:15.981908 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:15.981872    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:19.302138 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:19.302090    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:19.627778 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:19.627648    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:20.675963 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:20.675926    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:25.418199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:25.415842    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:25.919104 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:25.919074    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:29.181318 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:23:29.181281    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:23:30.177120 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.177089    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:30.608323 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.608282    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"]
Oct 31 01:23:30.612537 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.612521    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.734692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.734626    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.734952 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.734704    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.735049 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.734969    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rffp2\" (UniqueName: \"kubernetes.io/projected/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-kube-api-access-rffp2\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.735133 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.735120    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.735398 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.735149    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.735398 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.735339    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.735702 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.735591    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.735702 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.735663    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.735867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.735737    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.815335 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.815293    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"]
Oct 31 01:23:30.837535 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.837488    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.837719 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.837546    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.837719 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.837596    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.837719 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.837690    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.837912 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.837721    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.837912 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.837787    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rffp2\" (UniqueName: \"kubernetes.io/projected/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-kube-api-access-rffp2\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.837912 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.837838    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.837912 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.837863    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.838139 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.837921    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.838139 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.837948    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.838666 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.838641    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.839150 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.839125    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.839276 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.839231    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.839649 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.839620    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.843121 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.843093    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.858111 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.858079    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.894888 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.894825    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.894993 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.894979    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rffp2\" (UniqueName: \"kubernetes.io/projected/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-kube-api-access-rffp2\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:30.926968 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:30.926944    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:31.354162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:31.354130    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"]
Oct 31 01:23:31.355120 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:23:31.355093    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb725fb1_db73_4ca4_9dba_d4c2028f79e6.slice/crio-0c35a0c873cbb19318bd257ae1cce910c8e0b89a9f8dec2fb7f6623f3f7de5de WatchSource:0}: Error finding container 0c35a0c873cbb19318bd257ae1cce910c8e0b89a9f8dec2fb7f6623f3f7de5de: Status 404 returned error can't find the container with id 0c35a0c873cbb19318bd257ae1cce910c8e0b89a9f8dec2fb7f6623f3f7de5de
Oct 31 01:23:31.357652 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:31.357616    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:23:31.357765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:31.357695    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:23:31.357765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:31.357740    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:23:31.681726 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:31.681629    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9" event={"ID":"eb725fb1-db73-4ca4-9dba-d4c2028f79e6","Type":"ContainerStarted","Data":"2b943ade8183ca57c83f964924daef6fc10edf7eefe4923b0ea414af0069052e"}
Oct 31 01:23:31.681726 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:31.681698    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9" event={"ID":"eb725fb1-db73-4ca4-9dba-d4c2028f79e6","Type":"ContainerStarted","Data":"0c35a0c873cbb19318bd257ae1cce910c8e0b89a9f8dec2fb7f6623f3f7de5de"}
Oct 31 01:23:31.799905 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:31.799845    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9" podStartSLOduration=1.7998272690000001 podStartE2EDuration="1.799827269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:31.796821359 +0000 UTC m=+1621.486704523" watchObservedRunningTime="2026-10-31 01:23:31.799827269 +0000 UTC m=+1621.489710396"
Oct 31 01:23:31.927284 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:31.927252    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:32.928478 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:32.928438    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:32.929002 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:32.928495    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9" podUID="eb725fb1-db73-4ca4-9dba-d4c2028f79e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 01:23:33.928293 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:33.928257    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:33.928476 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:33.928312    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9" podUID="eb725fb1-db73-4ca4-9dba-d4c2028f79e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 01:23:34.928298 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:34.928260    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:34.928892 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:34.928860    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9" podUID="eb725fb1-db73-4ca4-9dba-d4c2028f79e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 01:23:34.930152 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:34.930109    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:35.779202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:35.776041    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:35.927333 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:35.927303    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:35.927502 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:35.927360    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9" podUID="eb725fb1-db73-4ca4-9dba-d4c2028f79e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 01:23:36.928011 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:36.927979    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:36.928447 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:36.928030    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9" podUID="eb725fb1-db73-4ca4-9dba-d4c2028f79e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 01:23:37.928125 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:37.928092    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:37.928528 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:37.928142    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9" podUID="eb725fb1-db73-4ca4-9dba-d4c2028f79e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 01:23:38.932012 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:38.931985    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:38.932464 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:38.932446    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:38.933221 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:38.933205    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:23:39.642736 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:39.642706    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:39.840263 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:39.840230    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:44.382044 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:44.381952    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:46.791371 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:46.787042    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:47.202278 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:47.202200    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:49.193053 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:49.193011    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:50.759823 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:50.759785    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:50.954875 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:50.954846    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:52.739764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:52.739725    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:53.458893 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:53.458861    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:54.206459 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:54.206429    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:54.404385 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:54.404349    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:54.595232 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:54.595185    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:54.896707 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:54.896627    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:59.181998 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:23:59.181962    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:23:59.249470 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:59.249387    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:59.501426 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:59.501344    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:59.694235 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:23:59.694207    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:02.679090 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:02.679048    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:02.853250 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:02.853166    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:04.304012 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:04.303977    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:09.155203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:09.154264    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:09.476720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:09.476629    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:13.888951 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:13.888874    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:18.575198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:18.575136    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:19.971755 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:19.971722    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:23.277533 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:23.277487    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:27.926075 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:27.926042    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:28.102420 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:28.102393    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:28.348901 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:28.348870    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:29.181347 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:24:29.181316    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:24:30.800470 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:30.800426    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:31.929096 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:31.929063    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:32.110401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:32.110366    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:32.980244 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:32.980213    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:35.265599 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:35.265569    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:36.130423 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:36.130394    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:37.507893 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:37.507865    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:38.224497 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:38.224463    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:39.343430 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:39.343396    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:39.519157 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:39.519123    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:40.197079 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:40.197046    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:43.073276 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:43.073245    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:43.258580 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:43.258544    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:43.506432 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:43.506353    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:48.092904 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:48.092874    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:53.096029 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:53.095995    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:57.801277 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:57.801239    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:59.182645 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:24:59.182609    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:24:59.820870 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:24:59.820838    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:02.583212 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:02.583160    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:07.430121 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:07.430089    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:09.852952 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:09.852922    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:10.453985 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:10.453867    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:10.716392 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:10.716312    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:12.318774 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:12.318747    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:14.574737 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:14.574704    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:15.782420 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:15.782386    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:17.433812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:17.433775    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:17.776266 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:17.776153    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:17.974044 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:17.974000    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:18.134566 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.134526    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"]
Oct 31 01:25:18.139458 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.139431    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"
Oct 31 01:25:18.194493 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.194460    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"]
Oct 31 01:25:18.239906 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.239863    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-02c8fd\""
Oct 31 01:25:18.240112 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.239987    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-02c8fd-dockercfg-jgv47\""
Oct 31 01:25:18.284059 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.284022    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830527e-e546-4295-b5be-6d6cf5aa5129-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v\" (UID: \"4830527e-e546-4295-b5be-6d6cf5aa5129\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"
Oct 31 01:25:18.284285 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.284169    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxz5n\" (UniqueName: \"kubernetes.io/projected/4830527e-e546-4295-b5be-6d6cf5aa5129-kube-api-access-lxz5n\") pod \"payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v\" (UID: \"4830527e-e546-4295-b5be-6d6cf5aa5129\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"
Oct 31 01:25:18.284445 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.284425    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"]
Oct 31 01:25:18.294577 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.294549    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"
Oct 31 01:25:18.347446 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.347413    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"]
Oct 31 01:25:18.385968 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.385885    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b48vk\" (UniqueName: \"kubernetes.io/projected/a5f5ee47-434e-438e-8537-7bca802f8658-kube-api-access-b48vk\") pod \"payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx\" (UID: \"a5f5ee47-434e-438e-8537-7bca802f8658\") " pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"
Oct 31 01:25:18.385968 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.385925    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f5ee47-434e-438e-8537-7bca802f8658-plugins-config-volume\") pod \"payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx\" (UID: \"a5f5ee47-434e-438e-8537-7bca802f8658\") " pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"
Oct 31 01:25:18.386163 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.386109    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830527e-e546-4295-b5be-6d6cf5aa5129-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v\" (UID: \"4830527e-e546-4295-b5be-6d6cf5aa5129\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"
Oct 31 01:25:18.386163 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.386152    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxz5n\" (UniqueName: \"kubernetes.io/projected/4830527e-e546-4295-b5be-6d6cf5aa5129-kube-api-access-lxz5n\") pod \"payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v\" (UID: \"4830527e-e546-4295-b5be-6d6cf5aa5129\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"
Oct 31 01:25:18.386827 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.386806    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830527e-e546-4295-b5be-6d6cf5aa5129-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v\" (UID: \"4830527e-e546-4295-b5be-6d6cf5aa5129\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"
Oct 31 01:25:18.443563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.443530    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxz5n\" (UniqueName: \"kubernetes.io/projected/4830527e-e546-4295-b5be-6d6cf5aa5129-kube-api-access-lxz5n\") pod \"payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v\" (UID: \"4830527e-e546-4295-b5be-6d6cf5aa5129\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"
Oct 31 01:25:18.457425 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.457394    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"
Oct 31 01:25:18.489014 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.488985    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b48vk\" (UniqueName: \"kubernetes.io/projected/a5f5ee47-434e-438e-8537-7bca802f8658-kube-api-access-b48vk\") pod \"payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx\" (UID: \"a5f5ee47-434e-438e-8537-7bca802f8658\") " pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"
Oct 31 01:25:18.489210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.489024    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f5ee47-434e-438e-8537-7bca802f8658-plugins-config-volume\") pod \"payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx\" (UID: \"a5f5ee47-434e-438e-8537-7bca802f8658\") " pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"
Oct 31 01:25:18.489884 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.489862    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f5ee47-434e-438e-8537-7bca802f8658-plugins-config-volume\") pod \"payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx\" (UID: \"a5f5ee47-434e-438e-8537-7bca802f8658\") " pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"
Oct 31 01:25:18.566221 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.565240    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b48vk\" (UniqueName: \"kubernetes.io/projected/a5f5ee47-434e-438e-8537-7bca802f8658-kube-api-access-b48vk\") pod \"payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx\" (UID: \"a5f5ee47-434e-438e-8537-7bca802f8658\") " pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"
Oct 31 01:25:18.616247 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.616214    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"
Oct 31 01:25:18.853210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:18.853161    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"]
Oct 31 01:25:19.001152 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:19.001124    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"]
Oct 31 01:25:19.002196 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:25:19.002151    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f5ee47_434e_438e_8537_7bca802f8658.slice/crio-062b468f244098a20425dfb1a722817a6e165f962b4c69a7503c9219f68fb578 WatchSource:0}: Error finding container 062b468f244098a20425dfb1a722817a6e165f962b4c69a7503c9219f68fb578: Status 404 returned error can't find the container with id 062b468f244098a20425dfb1a722817a6e165f962b4c69a7503c9219f68fb578
Oct 31 01:25:19.449084 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:19.449051    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx" event={"ID":"a5f5ee47-434e-438e-8537-7bca802f8658","Type":"ContainerStarted","Data":"c28e7128ced74f0ca38893b4b1811d2f2fc84f7cdc0afd3e913f87443bed9294"}
Oct 31 01:25:19.449557 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:19.449109    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx" event={"ID":"a5f5ee47-434e-438e-8537-7bca802f8658","Type":"ContainerStarted","Data":"062b468f244098a20425dfb1a722817a6e165f962b4c69a7503c9219f68fb578"}
Oct 31 01:25:19.449557 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:19.449267    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"
Oct 31 01:25:19.450782 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:19.450757    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v" event={"ID":"4830527e-e546-4295-b5be-6d6cf5aa5129","Type":"ContainerStarted","Data":"8e67fd957893693ca3557edce6af5296cb1a6b945e79bc8774b7df7385c51945"}
Oct 31 01:25:19.450902 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:19.450800    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v" event={"ID":"4830527e-e546-4295-b5be-6d6cf5aa5129","Type":"ContainerStarted","Data":"9714823d8cb915f7fa93e835983ae8c7ecef2c9a5faec0b4d10c384450e07c28"}
Oct 31 01:25:19.450902 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:19.450833    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"
Oct 31 01:25:19.562224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:19.562145    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx" podStartSLOduration=1.5621313479999999 podStartE2EDuration="1.562131348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:19.561266415 +0000 UTC m=+1729.251149543" watchObservedRunningTime="2026-10-31 01:25:19.562131348 +0000 UTC m=+1729.252014473"
Oct 31 01:25:20.407383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:20.407348    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:22.483250 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:22.483135    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:24.523553 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:24.523503    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:24.727732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:24.727700    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:27.355957 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:27.355923    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:29.181795 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:25:29.181758    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:25:29.226935 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:29.226905    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:30.460263 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:30.460235    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"
Oct 31 01:25:30.460663 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:30.460306    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"
Oct 31 01:25:30.576369 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:30.576323    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v" podStartSLOduration=12.576309287 podStartE2EDuration="12.576309287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:19.677417101 +0000 UTC m=+1729.367300228" watchObservedRunningTime="2026-10-31 01:25:30.576309287 +0000 UTC m=+1740.266192413"
Oct 31 01:25:32.929231 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:32.929199    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:36.840447 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:36.840405    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:37.020684 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:37.020651    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:37.721136 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:37.721103    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:38.832660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:38.832600    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:39.040126 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:39.040096    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:40.562759 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:40.562732    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:42.747334 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:42.745928    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:43.005586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:43.005493    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:43.213713 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:43.213678    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:46.549484 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:46.549456    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:47.959132 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:47.959096    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:48.140976 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.140928    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:48.341041 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.341003    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"]
Oct 31 01:25:48.346004 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.345981    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"
Oct 31 01:25:48.397351 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.397313    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"]
Oct 31 01:25:48.439550 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.439516    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxsfw\" (UniqueName: \"kubernetes.io/projected/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe-kube-api-access-bxsfw\") pod \"payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l\" (UID: \"e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"
Oct 31 01:25:48.439723 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.439578    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l\" (UID: \"e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"
Oct 31 01:25:48.440617 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.440583    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-987f2f\""
Oct 31 01:25:48.441216 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.441160    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-987f2f-dockercfg-x5v55\""
Oct 31 01:25:48.475336 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.475307    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"]
Oct 31 01:25:48.479798 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.479766    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"
Oct 31 01:25:48.539780 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.539742    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"]
Oct 31 01:25:48.540847 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.540800    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxsfw\" (UniqueName: \"kubernetes.io/projected/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe-kube-api-access-bxsfw\") pod \"payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l\" (UID: \"e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"
Oct 31 01:25:48.541045 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.540879    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l\" (UID: \"e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"
Oct 31 01:25:48.541045 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.540923    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae7c551-8aa2-4728-84ab-c12f662e8387-plugins-config-volume\") pod \"payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp\" (UID: \"6ae7c551-8aa2-4728-84ab-c12f662e8387\") " pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"
Oct 31 01:25:48.546401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.541161    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd6fm\" (UniqueName: \"kubernetes.io/projected/6ae7c551-8aa2-4728-84ab-c12f662e8387-kube-api-access-zd6fm\") pod \"payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp\" (UID: \"6ae7c551-8aa2-4728-84ab-c12f662e8387\") " pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"
Oct 31 01:25:48.546401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.542057    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l\" (UID: \"e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"
Oct 31 01:25:48.601286 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.601198    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxsfw\" (UniqueName: \"kubernetes.io/projected/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe-kube-api-access-bxsfw\") pod \"payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l\" (UID: \"e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"
Oct 31 01:25:48.642260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.642231    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae7c551-8aa2-4728-84ab-c12f662e8387-plugins-config-volume\") pod \"payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp\" (UID: \"6ae7c551-8aa2-4728-84ab-c12f662e8387\") " pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"
Oct 31 01:25:48.642450 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.642371    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd6fm\" (UniqueName: \"kubernetes.io/projected/6ae7c551-8aa2-4728-84ab-c12f662e8387-kube-api-access-zd6fm\") pod \"payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp\" (UID: \"6ae7c551-8aa2-4728-84ab-c12f662e8387\") " pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"
Oct 31 01:25:48.643226 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.643162    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae7c551-8aa2-4728-84ab-c12f662e8387-plugins-config-volume\") pod \"payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp\" (UID: \"6ae7c551-8aa2-4728-84ab-c12f662e8387\") " pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"
Oct 31 01:25:48.660908 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.660876    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"
Oct 31 01:25:48.713236 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.707377    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd6fm\" (UniqueName: \"kubernetes.io/projected/6ae7c551-8aa2-4728-84ab-c12f662e8387-kube-api-access-zd6fm\") pod \"payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp\" (UID: \"6ae7c551-8aa2-4728-84ab-c12f662e8387\") " pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"
Oct 31 01:25:48.799241 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:48.798399    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"
Oct 31 01:25:49.072273 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:49.072245    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"]
Oct 31 01:25:49.073303 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:25:49.073278    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56d5228_1d0f_4fd2_b0e0_f6f475ae2cbe.slice/crio-3ca910f31cf79154b059a620fdcef1b270aa83ecb9e1eb77c9cda6726310475f WatchSource:0}: Error finding container 3ca910f31cf79154b059a620fdcef1b270aa83ecb9e1eb77c9cda6726310475f: Status 404 returned error can't find the container with id 3ca910f31cf79154b059a620fdcef1b270aa83ecb9e1eb77c9cda6726310475f
Oct 31 01:25:49.215415 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:49.215390    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"]
Oct 31 01:25:49.217437 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:25:49.217410    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae7c551_8aa2_4728_84ab_c12f662e8387.slice/crio-89ee120a3192093dbf4bf66c591d26ee8416d633ac48f194ba45017674f840e5 WatchSource:0}: Error finding container 89ee120a3192093dbf4bf66c591d26ee8416d633ac48f194ba45017674f840e5: Status 404 returned error can't find the container with id 89ee120a3192093dbf4bf66c591d26ee8416d633ac48f194ba45017674f840e5
Oct 31 01:25:49.669139 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:49.669103    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp" event={"ID":"6ae7c551-8aa2-4728-84ab-c12f662e8387","Type":"ContainerStarted","Data":"6c5e2c85883bae029cef811cf937dac3ef6eb2cc8bb451097fb2fc75f8ade432"}
Oct 31 01:25:49.669355 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:49.669200    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp" event={"ID":"6ae7c551-8aa2-4728-84ab-c12f662e8387","Type":"ContainerStarted","Data":"89ee120a3192093dbf4bf66c591d26ee8416d633ac48f194ba45017674f840e5"}
Oct 31 01:25:49.669420 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:49.669364    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"
Oct 31 01:25:49.670950 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:49.670921    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l" event={"ID":"e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe","Type":"ContainerStarted","Data":"a634b54093a6a4d69e898bdb3b8b377c4c1bddf2ec3d69dc4760bbd8bbcce04f"}
Oct 31 01:25:49.671061 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:49.670965    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l" event={"ID":"e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe","Type":"ContainerStarted","Data":"3ca910f31cf79154b059a620fdcef1b270aa83ecb9e1eb77c9cda6726310475f"}
Oct 31 01:25:49.671061 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:49.670993    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"
Oct 31 01:25:49.789782 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:49.789716    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp" podStartSLOduration=1.789702688 podStartE2EDuration="1.789702688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:49.78554602 +0000 UTC m=+1759.475429157" watchObservedRunningTime="2026-10-31 01:25:49.789702688 +0000 UTC m=+1759.479585815"
Oct 31 01:25:49.908031 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:49.907964    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l" podStartSLOduration=1.9079478970000001 podStartE2EDuration="1.907947897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:49.899155099 +0000 UTC m=+1759.589038227" watchObservedRunningTime="2026-10-31 01:25:49.907947897 +0000 UTC m=+1759.597831024"
Oct 31 01:25:53.242201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:53.240527    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:58.446342 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:25:58.446295    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:59.183078 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:25:59.183037    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:26:00.598452 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:00.598420    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"]
Oct 31 01:26:00.679633 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:00.679607    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"
Oct 31 01:26:00.679802 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:00.679727    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"
Oct 31 01:26:01.022845 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:01.022767    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:01.206869 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:01.206839    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:03.231599 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:03.231548    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:03.411021 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:03.410979    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:03.585737 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:03.585700    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:05.598985 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:05.598930    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg" podUID="640ee4f4-6d6d-49a8-b2b1-686e39623b85" containerName="payload-processing" containerID="cri-o://747b54820c9ce8a77bc870e76de85127cb12c55d399fc2b77b4d2c92bb60b830" gracePeriod=25
Oct 31 01:26:05.776187 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:05.776153    4476 generic.go:356] "Generic (PLEG): container finished" podID="640ee4f4-6d6d-49a8-b2b1-686e39623b85" containerID="747b54820c9ce8a77bc870e76de85127cb12c55d399fc2b77b4d2c92bb60b830" exitCode=0
Oct 31 01:26:05.776347 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:05.776211    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg" event={"ID":"640ee4f4-6d6d-49a8-b2b1-686e39623b85","Type":"ContainerDied","Data":"747b54820c9ce8a77bc870e76de85127cb12c55d399fc2b77b4d2c92bb60b830"}
Oct 31 01:26:05.885874 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:05.885851    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"
Oct 31 01:26:05.894322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:05.894296    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/640ee4f4-6d6d-49a8-b2b1-686e39623b85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640ee4f4-6d6d-49a8-b2b1-686e39623b85-plugins-config-volume\") pod \"640ee4f4-6d6d-49a8-b2b1-686e39623b85\" (UID: \"640ee4f4-6d6d-49a8-b2b1-686e39623b85\") "
Oct 31 01:26:05.894424 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:05.894337    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/640ee4f4-6d6d-49a8-b2b1-686e39623b85-kube-api-access-9rchs\" (UniqueName: \"kubernetes.io/projected/640ee4f4-6d6d-49a8-b2b1-686e39623b85-kube-api-access-9rchs\") pod \"640ee4f4-6d6d-49a8-b2b1-686e39623b85\" (UID: \"640ee4f4-6d6d-49a8-b2b1-686e39623b85\") "
Oct 31 01:26:05.894605 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:05.894586    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/640ee4f4-6d6d-49a8-b2b1-686e39623b85-plugins-config-volume" pod "640ee4f4-6d6d-49a8-b2b1-686e39623b85" (UID: "640ee4f4-6d6d-49a8-b2b1-686e39623b85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:05.894810 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:05.894796    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640ee4f4-6d6d-49a8-b2b1-686e39623b85-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:05.896543 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:05.896518    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/640ee4f4-6d6d-49a8-b2b1-686e39623b85-kube-api-access-9rchs" pod "640ee4f4-6d6d-49a8-b2b1-686e39623b85" (UID: "640ee4f4-6d6d-49a8-b2b1-686e39623b85"). InnerVolumeSpecName "kube-api-access-9rchs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:05.995741 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:05.995709    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rchs\" (UniqueName: \"kubernetes.io/projected/640ee4f4-6d6d-49a8-b2b1-686e39623b85-kube-api-access-9rchs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:06.784350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:06.784317    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg" event={"ID":"640ee4f4-6d6d-49a8-b2b1-686e39623b85","Type":"ContainerDied","Data":"448c319129dfc713a7269b0c8a954a6160200c72be7ce4aaba4ef1b4b91115c0"}
Oct 31 01:26:06.784787 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:06.784331    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"
Oct 31 01:26:06.784787 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:06.784414    4476 scope.go:122] "RemoveContainer" containerID="747b54820c9ce8a77bc870e76de85127cb12c55d399fc2b77b4d2c92bb60b830"
Oct 31 01:26:06.947819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:06.947782    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"]
Oct 31 01:26:06.997117 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:06.997081    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4bcda6-64d5569cf-mjvdg"]
Oct 31 01:26:08.322223 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:08.322163    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:08.455241 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:08.455203    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="640ee4f4-6d6d-49a8-b2b1-686e39623b85" path="/var/lib/kubelet/pods/640ee4f4-6d6d-49a8-b2b1-686e39623b85/volumes"
Oct 31 01:26:10.188342 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:10.188307    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:13.063396 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:13.063361    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:17.775429 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:17.775392    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:19.815102 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:19.815072    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:22.724491 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:22.724458    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:27.755546 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:27.755509    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:28.292246 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:28.292203    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:29.052546 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:29.052512    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:29.182258 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:26:29.182221    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:26:30.070307 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:30.070263    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:30.311986 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:30.311954    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:30.658011 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:30.657979    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:26:30.681829 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:30.681806    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:26:30.868247 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:30.868195    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"]
Oct 31 01:26:30.868502 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:30.868478    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg" podUID="5245c2fd-f28f-4965-a210-bc0cf9a10376" containerName="istio-proxy" containerID="cri-o://d8fa899b917948c742825f40c6da2bd3c5f03926bb8c5b14df82dcc805e77bf0" gracePeriod=30
Oct 31 01:26:32.444252 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:32.444195    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:32.619709 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:32.619676    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:32.800023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:32.799993    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:34.439595 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:34.439556    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:34.769369 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:34.769287    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:35.486250 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:35.486215    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:26:35.486772 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:35.486287    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg" podUID="5245c2fd-f28f-4965-a210-bc0cf9a10376" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:26:36.018333 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.018284    4476 generic.go:356] "Generic (PLEG): container finished" podID="5245c2fd-f28f-4965-a210-bc0cf9a10376" containerID="d8fa899b917948c742825f40c6da2bd3c5f03926bb8c5b14df82dcc805e77bf0" exitCode=0
Oct 31 01:26:36.018508 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.018453    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg" event={"ID":"5245c2fd-f28f-4965-a210-bc0cf9a10376","Type":"ContainerDied","Data":"d8fa899b917948c742825f40c6da2bd3c5f03926bb8c5b14df82dcc805e77bf0"}
Oct 31 01:26:36.166990 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.166966    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:26:36.326036 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.326006    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-envoy\") pod \"5245c2fd-f28f-4965-a210-bc0cf9a10376\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") "
Oct 31 01:26:36.326224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.326094    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-credential-socket\") pod \"5245c2fd-f28f-4965-a210-bc0cf9a10376\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") "
Oct 31 01:26:36.326224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.326150    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-workload-socket\") pod \"5245c2fd-f28f-4965-a210-bc0cf9a10376\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") "
Oct 31 01:26:36.326333 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.326239    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-podinfo\") pod \"5245c2fd-f28f-4965-a210-bc0cf9a10376\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") "
Oct 31 01:26:36.326333 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.326281    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5245c2fd-f28f-4965-a210-bc0cf9a10376-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5245c2fd-f28f-4965-a210-bc0cf9a10376-istiod-ca-cert\") pod \"5245c2fd-f28f-4965-a210-bc0cf9a10376\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") "
Oct 31 01:26:36.326439 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.326374    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-token\" (UniqueName: \"kubernetes.io/projected/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-token\") pod \"5245c2fd-f28f-4965-a210-bc0cf9a10376\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") "
Oct 31 01:26:36.326439 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.326400    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-workload-certs\") pod \"5245c2fd-f28f-4965-a210-bc0cf9a10376\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") "
Oct 31 01:26:36.326550 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.326438    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5245c2fd-f28f-4965-a210-bc0cf9a10376-kube-api-access-cxfvf\" (UniqueName: \"kubernetes.io/projected/5245c2fd-f28f-4965-a210-bc0cf9a10376-kube-api-access-cxfvf\") pod \"5245c2fd-f28f-4965-a210-bc0cf9a10376\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") "
Oct 31 01:26:36.326550 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.326444    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-credential-socket" pod "5245c2fd-f28f-4965-a210-bc0cf9a10376" (UID: "5245c2fd-f28f-4965-a210-bc0cf9a10376"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:36.326550 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.326469    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-data\") pod \"5245c2fd-f28f-4965-a210-bc0cf9a10376\" (UID: \"5245c2fd-f28f-4965-a210-bc0cf9a10376\") "
Oct 31 01:26:36.326709 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.326668    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-workload-certs" pod "5245c2fd-f28f-4965-a210-bc0cf9a10376" (UID: "5245c2fd-f28f-4965-a210-bc0cf9a10376"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:36.326768 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.326700    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5245c2fd-f28f-4965-a210-bc0cf9a10376-istiod-ca-cert" pod "5245c2fd-f28f-4965-a210-bc0cf9a10376" (UID: "5245c2fd-f28f-4965-a210-bc0cf9a10376"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:36.326957 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.326936    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-data" pod "5245c2fd-f28f-4965-a210-bc0cf9a10376" (UID: "5245c2fd-f28f-4965-a210-bc0cf9a10376"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:36.327107 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.326986    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-workload-socket" pod "5245c2fd-f28f-4965-a210-bc0cf9a10376" (UID: "5245c2fd-f28f-4965-a210-bc0cf9a10376"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:36.327233 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.327158    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:36.327307 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.327242    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:36.327307 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.327259    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5245c2fd-f28f-4965-a210-bc0cf9a10376-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:36.328569 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.328540    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-podinfo" pod "5245c2fd-f28f-4965-a210-bc0cf9a10376" (UID: "5245c2fd-f28f-4965-a210-bc0cf9a10376"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:26:36.328668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.328565    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-envoy" pod "5245c2fd-f28f-4965-a210-bc0cf9a10376" (UID: "5245c2fd-f28f-4965-a210-bc0cf9a10376"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:36.329301 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.329279    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5245c2fd-f28f-4965-a210-bc0cf9a10376-kube-api-access-cxfvf" pod "5245c2fd-f28f-4965-a210-bc0cf9a10376" (UID: "5245c2fd-f28f-4965-a210-bc0cf9a10376"). InnerVolumeSpecName "kube-api-access-cxfvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:36.329392 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.329316    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-token" pod "5245c2fd-f28f-4965-a210-bc0cf9a10376" (UID: "5245c2fd-f28f-4965-a210-bc0cf9a10376"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:36.428819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.428747    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:36.428819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.428774    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxfvf\" (UniqueName: \"kubernetes.io/projected/5245c2fd-f28f-4965-a210-bc0cf9a10376-kube-api-access-cxfvf\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:36.428819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.428786    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:36.428819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.428795    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:36.428819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.428803    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245c2fd-f28f-4965-a210-bc0cf9a10376-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:36.428819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:36.428812    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5245c2fd-f28f-4965-a210-bc0cf9a10376-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:37.025731 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:37.025701    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"
Oct 31 01:26:37.025731 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:37.025726    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg" event={"ID":"5245c2fd-f28f-4965-a210-bc0cf9a10376","Type":"ContainerDied","Data":"c0a4a309ebad08fe63be732f2490fa45c393b59717ece72f6287f3534efb4db0"}
Oct 31 01:26:37.026258 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:37.025783    4476 scope.go:122] "RemoveContainer" containerID="d8fa899b917948c742825f40c6da2bd3c5f03926bb8c5b14df82dcc805e77bf0"
Oct 31 01:26:37.239050 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:37.239022    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"]
Oct 31 01:26:37.288105 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:37.288079    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-kcghg"]
Oct 31 01:26:37.696387 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:37.696250    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:37.903767 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:37.903728    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:38.454790 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:38.454762    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5245c2fd-f28f-4965-a210-bc0cf9a10376" path="/var/lib/kubelet/pods/5245c2fd-f28f-4965-a210-bc0cf9a10376/volumes"
Oct 31 01:26:40.984297 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:40.984255    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:42.014404 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.014371    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"]
Oct 31 01:26:42.014869 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.014562    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd" podUID="46f84534-7aab-4133-8f6c-a6b2fa4983fb" containerName="maas-api" containerID="cri-o://109020bf8ccfe7ec3b9fc309b73cc4502ec7f0133fa442b061e36bcefe256a4d" gracePeriod=30
Oct 31 01:26:42.189955 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.189924    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:42.310042 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.310004    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:26:42.427644 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.427614    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46f84534-7aab-4133-8f6c-a6b2fa4983fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f84534-7aab-4133-8f6c-a6b2fa4983fb-openshift-service-ca\") pod \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") "
Oct 31 01:26:42.427824 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.427725    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-maas-api-tls\") pod \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") "
Oct 31 01:26:42.427906 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.427886    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f84534-7aab-4133-8f6c-a6b2fa4983fb-kube-api-access-kxswt\" (UniqueName: \"kubernetes.io/projected/46f84534-7aab-4133-8f6c-a6b2fa4983fb-kube-api-access-kxswt\") pod \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") "
Oct 31 01:26:42.428022 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.428001    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-metrics-certs\") pod \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\" (UID: \"46f84534-7aab-4133-8f6c-a6b2fa4983fb\") "
Oct 31 01:26:42.428444 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.428414    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f84534-7aab-4133-8f6c-a6b2fa4983fb-openshift-service-ca" pod "46f84534-7aab-4133-8f6c-a6b2fa4983fb" (UID: "46f84534-7aab-4133-8f6c-a6b2fa4983fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:42.428707 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.428685    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f84534-7aab-4133-8f6c-a6b2fa4983fb-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:42.430951 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.430929    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-metrics-certs" pod "46f84534-7aab-4133-8f6c-a6b2fa4983fb" (UID: "46f84534-7aab-4133-8f6c-a6b2fa4983fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:26:42.431466 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.431443    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f84534-7aab-4133-8f6c-a6b2fa4983fb-kube-api-access-kxswt" pod "46f84534-7aab-4133-8f6c-a6b2fa4983fb" (UID: "46f84534-7aab-4133-8f6c-a6b2fa4983fb"). InnerVolumeSpecName "kube-api-access-kxswt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:42.431550 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.431503    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-maas-api-tls" pod "46f84534-7aab-4133-8f6c-a6b2fa4983fb" (UID: "46f84534-7aab-4133-8f6c-a6b2fa4983fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:26:42.529694 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.529666    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:42.529841 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.529698    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f84534-7aab-4133-8f6c-a6b2fa4983fb-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:42.529841 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:42.529710    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxswt\" (UniqueName: \"kubernetes.io/projected/46f84534-7aab-4133-8f6c-a6b2fa4983fb-kube-api-access-kxswt\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:43.138007 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:43.137958    4476 generic.go:356] "Generic (PLEG): container finished" podID="46f84534-7aab-4133-8f6c-a6b2fa4983fb" containerID="109020bf8ccfe7ec3b9fc309b73cc4502ec7f0133fa442b061e36bcefe256a4d" exitCode=0
Oct 31 01:26:43.138585 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:43.138094    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd" event={"ID":"46f84534-7aab-4133-8f6c-a6b2fa4983fb","Type":"ContainerDied","Data":"109020bf8ccfe7ec3b9fc309b73cc4502ec7f0133fa442b061e36bcefe256a4d"}
Oct 31 01:26:43.138585 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:43.138147    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd" event={"ID":"46f84534-7aab-4133-8f6c-a6b2fa4983fb","Type":"ContainerDied","Data":"3d2dd3909c8afecc48b115217499b8273181e26326dbdc77aa1aa6debbd66be6"}
Oct 31 01:26:43.138585 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:43.138197    4476 scope.go:122] "RemoveContainer" containerID="109020bf8ccfe7ec3b9fc309b73cc4502ec7f0133fa442b061e36bcefe256a4d"
Oct 31 01:26:43.138585 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:43.138360    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"
Oct 31 01:26:43.178111 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:43.178084    4476 scope.go:122] "RemoveContainer" containerID="109020bf8ccfe7ec3b9fc309b73cc4502ec7f0133fa442b061e36bcefe256a4d"
Oct 31 01:26:43.178540 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:26:43.178510    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"109020bf8ccfe7ec3b9fc309b73cc4502ec7f0133fa442b061e36bcefe256a4d\": container with ID starting with 109020bf8ccfe7ec3b9fc309b73cc4502ec7f0133fa442b061e36bcefe256a4d not found: ID does not exist" containerID="109020bf8ccfe7ec3b9fc309b73cc4502ec7f0133fa442b061e36bcefe256a4d"
Oct 31 01:26:43.178760 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:43.178722    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"109020bf8ccfe7ec3b9fc309b73cc4502ec7f0133fa442b061e36bcefe256a4d"} err="failed to get container status \"109020bf8ccfe7ec3b9fc309b73cc4502ec7f0133fa442b061e36bcefe256a4d\": rpc error: code = NotFound desc = could not find container \"109020bf8ccfe7ec3b9fc309b73cc4502ec7f0133fa442b061e36bcefe256a4d\": container with ID starting with 109020bf8ccfe7ec3b9fc309b73cc4502ec7f0133fa442b061e36bcefe256a4d not found: ID does not exist"
Oct 31 01:26:43.303263 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:43.303233    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"]
Oct 31 01:26:43.352202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:43.352149    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f259852e-56d9866d6-qsqqd"]
Oct 31 01:26:43.760585 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:43.760550    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:43.968934 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:43.968906    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:44.161663 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:44.161633    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:44.455388 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:44.455303    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46f84534-7aab-4133-8f6c-a6b2fa4983fb" path="/var/lib/kubelet/pods/46f84534-7aab-4133-8f6c-a6b2fa4983fb/volumes"
Oct 31 01:26:44.840502 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:44.840472    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"]
Oct 31 01:26:44.840675 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:44.840661    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" podUID="1c75060f-8290-4054-826e-c0044394f5d5" containerName="istio-proxy" containerID="cri-o://07d7bf5ab9e27ca453cde71f42f3d1d273d3fba0a77ac5df80462aa324adf161" gracePeriod=30
Oct 31 01:26:46.110710 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:46.110681    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"]
Oct 31 01:26:46.111085 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:46.110885    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7" podUID="2a84b045-58f1-43bf-b02b-6ee4c9a62725" containerName="istio-proxy" containerID="cri-o://bb14bd08eb6fa6ae42de87d7d657a2e25aa5dc27f59777916da8b7367a7200d1" gracePeriod=30
Oct 31 01:26:46.165163 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:46.165134    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"]
Oct 31 01:26:46.165432 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:46.165408    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7" podUID="bc044185-a187-4f47-92ee-972bd7794f22" containerName="istio-proxy" containerID="cri-o://41355714b0b6e7f323fc30dcce040dbd85d6c85fd06f0fddae8cf6fe71471418" gracePeriod=30
Oct 31 01:26:47.378424 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:47.378394    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"]
Oct 31 01:26:47.378841 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:47.378608    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7" podUID="fb63a95e-790b-425c-bb97-9925b8f20a49" containerName="istio-proxy" containerID="cri-o://53bfa36de2857038f68263e6b016e60a9e1f3342aaf1fd7038c0f932d31feec0" gracePeriod=30
Oct 31 01:26:47.428400 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:47.428359    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"]
Oct 31 01:26:47.428654 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:47.428608    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerName="istio-proxy" containerID="cri-o://041e5e9ea3188212b09d71810fe253fa64f4182925545d8f4caf5bd609a847a9" gracePeriod=30
Oct 31 01:26:47.433263 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:47.433235    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"]
Oct 31 01:26:47.433486 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:47.433464    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6" podUID="689153b0-2780-48a8-b936-7b96ba9ca8b1" containerName="istio-proxy" containerID="cri-o://834a22d5c34b838ae06cd9e04954832bc73bb53e54e8779d13809b00229a07b8" gracePeriod=30
Oct 31 01:26:47.435774 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:47.435750    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"]
Oct 31 01:26:47.435947 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:47.435911    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn" podUID="a97eb81c-138f-42d7-a47e-a19504d68e7c" containerName="istio-proxy" containerID="cri-o://512047f077b2b2a616a05c7cea5f61043cddf37c2e03a997234b67031f968ef9" gracePeriod=30
Oct 31 01:26:48.453950 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:48.453902    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:48.640894 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:48.640862    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:48.886459 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:48.886425    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:50.122995 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.122929    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:26:50.184271 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.184240    4476 generic.go:356] "Generic (PLEG): container finished" podID="1c75060f-8290-4054-826e-c0044394f5d5" containerID="07d7bf5ab9e27ca453cde71f42f3d1d273d3fba0a77ac5df80462aa324adf161" exitCode=0
Oct 31 01:26:50.184438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.184287    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"
Oct 31 01:26:50.184438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.184314    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" event={"ID":"1c75060f-8290-4054-826e-c0044394f5d5","Type":"ContainerDied","Data":"07d7bf5ab9e27ca453cde71f42f3d1d273d3fba0a77ac5df80462aa324adf161"}
Oct 31 01:26:50.184438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.184388    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f" event={"ID":"1c75060f-8290-4054-826e-c0044394f5d5","Type":"ContainerDied","Data":"a58bbd28bbf5b243e166c9a782d28a7e7e4bb1927a0ee9d58a1902f126e42698"}
Oct 31 01:26:50.184438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.184410    4476 scope.go:122] "RemoveContainer" containerID="07d7bf5ab9e27ca453cde71f42f3d1d273d3fba0a77ac5df80462aa324adf161"
Oct 31 01:26:50.193683 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.193665    4476 scope.go:122] "RemoveContainer" containerID="07d7bf5ab9e27ca453cde71f42f3d1d273d3fba0a77ac5df80462aa324adf161"
Oct 31 01:26:50.193947 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:26:50.193929    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d7bf5ab9e27ca453cde71f42f3d1d273d3fba0a77ac5df80462aa324adf161\": container with ID starting with 07d7bf5ab9e27ca453cde71f42f3d1d273d3fba0a77ac5df80462aa324adf161 not found: ID does not exist" containerID="07d7bf5ab9e27ca453cde71f42f3d1d273d3fba0a77ac5df80462aa324adf161"
Oct 31 01:26:50.193986 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.193956    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d7bf5ab9e27ca453cde71f42f3d1d273d3fba0a77ac5df80462aa324adf161"} err="failed to get container status \"07d7bf5ab9e27ca453cde71f42f3d1d273d3fba0a77ac5df80462aa324adf161\": rpc error: code = NotFound desc = could not find container \"07d7bf5ab9e27ca453cde71f42f3d1d273d3fba0a77ac5df80462aa324adf161\": container with ID starting with 07d7bf5ab9e27ca453cde71f42f3d1d273d3fba0a77ac5df80462aa324adf161 not found: ID does not exist"
Oct 31 01:26:50.233897 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.233873    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-workload-certs\") pod \"1c75060f-8290-4054-826e-c0044394f5d5\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") "
Oct 31 01:26:50.234030 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.233923    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-credential-socket\") pod \"1c75060f-8290-4054-826e-c0044394f5d5\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") "
Oct 31 01:26:50.234030 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.233947    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c75060f-8290-4054-826e-c0044394f5d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c75060f-8290-4054-826e-c0044394f5d5-istio-podinfo\") pod \"1c75060f-8290-4054-826e-c0044394f5d5\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") "
Oct 31 01:26:50.234030 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.233990    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c75060f-8290-4054-826e-c0044394f5d5-kube-api-access-tlt5d\" (UniqueName: \"kubernetes.io/projected/1c75060f-8290-4054-826e-c0044394f5d5-kube-api-access-tlt5d\") pod \"1c75060f-8290-4054-826e-c0044394f5d5\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") "
Oct 31 01:26:50.234225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234057    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c75060f-8290-4054-826e-c0044394f5d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c75060f-8290-4054-826e-c0044394f5d5-istiod-ca-cert\") pod \"1c75060f-8290-4054-826e-c0044394f5d5\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") "
Oct 31 01:26:50.234225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234103    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-istio-data\") pod \"1c75060f-8290-4054-826e-c0044394f5d5\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") "
Oct 31 01:26:50.234225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234127    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-workload-certs" pod "1c75060f-8290-4054-826e-c0044394f5d5" (UID: "1c75060f-8290-4054-826e-c0044394f5d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:50.234225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234188    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-workload-socket\") pod \"1c75060f-8290-4054-826e-c0044394f5d5\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") "
Oct 31 01:26:50.234441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234228    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-credential-socket" pod "1c75060f-8290-4054-826e-c0044394f5d5" (UID: "1c75060f-8290-4054-826e-c0044394f5d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:50.234441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234288    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c75060f-8290-4054-826e-c0044394f5d5-istio-token\" (UniqueName: \"kubernetes.io/projected/1c75060f-8290-4054-826e-c0044394f5d5-istio-token\") pod \"1c75060f-8290-4054-826e-c0044394f5d5\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") "
Oct 31 01:26:50.234441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234320    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-istio-envoy\") pod \"1c75060f-8290-4054-826e-c0044394f5d5\" (UID: \"1c75060f-8290-4054-826e-c0044394f5d5\") "
Oct 31 01:26:50.234441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234388    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-istio-data" pod "1c75060f-8290-4054-826e-c0044394f5d5" (UID: "1c75060f-8290-4054-826e-c0044394f5d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:50.234700 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234460    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c75060f-8290-4054-826e-c0044394f5d5-istiod-ca-cert" pod "1c75060f-8290-4054-826e-c0044394f5d5" (UID: "1c75060f-8290-4054-826e-c0044394f5d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:50.234700 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234574    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-workload-socket" pod "1c75060f-8290-4054-826e-c0044394f5d5" (UID: "1c75060f-8290-4054-826e-c0044394f5d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:50.234858 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234837    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:50.234927 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234861    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:50.234927 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234871    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c75060f-8290-4054-826e-c0044394f5d5-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:50.234927 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234879    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:50.234927 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.234887    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:50.236345 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.236318    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c75060f-8290-4054-826e-c0044394f5d5-istio-podinfo" pod "1c75060f-8290-4054-826e-c0044394f5d5" (UID: "1c75060f-8290-4054-826e-c0044394f5d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:26:50.236442 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.236372    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c75060f-8290-4054-826e-c0044394f5d5-kube-api-access-tlt5d" pod "1c75060f-8290-4054-826e-c0044394f5d5" (UID: "1c75060f-8290-4054-826e-c0044394f5d5"). InnerVolumeSpecName "kube-api-access-tlt5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:50.236570 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.236549    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c75060f-8290-4054-826e-c0044394f5d5-istio-token" pod "1c75060f-8290-4054-826e-c0044394f5d5" (UID: "1c75060f-8290-4054-826e-c0044394f5d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:50.236661 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.236644    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-istio-envoy" pod "1c75060f-8290-4054-826e-c0044394f5d5" (UID: "1c75060f-8290-4054-826e-c0044394f5d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:50.335744 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.335718    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c75060f-8290-4054-826e-c0044394f5d5-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:50.335744 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.335744    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlt5d\" (UniqueName: \"kubernetes.io/projected/1c75060f-8290-4054-826e-c0044394f5d5-kube-api-access-tlt5d\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:50.335922 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.335755    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c75060f-8290-4054-826e-c0044394f5d5-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:50.335922 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.335764    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c75060f-8290-4054-826e-c0044394f5d5-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:50.601638 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.601608    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"]
Oct 31 01:26:50.609532 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.609498    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-xkf7f"]
Oct 31 01:26:50.657944 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.657915    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:26:50.658063 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.657973    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:26:50.958876 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.958845    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:26:50.959041 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:50.958901    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7" podUID="fb63a95e-790b-425c-bb97-9925b8f20a49" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:26:51.048202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.048129    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:26:51.048361 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.048221    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7" podUID="2a84b045-58f1-43bf-b02b-6ee4c9a62725" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:26:51.464844 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.464818    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:26:51.498188 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.498148    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:26:51.525848 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.525786    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:26:51.525848 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.525835    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6" podUID="689153b0-2780-48a8-b936-7b96ba9ca8b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:26:51.549386 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.549361    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a84b045-58f1-43bf-b02b-6ee4c9a62725-kube-api-access-2vhfp\" (UniqueName: \"kubernetes.io/projected/2a84b045-58f1-43bf-b02b-6ee4c9a62725-kube-api-access-2vhfp\") pod \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") "
Oct 31 01:26:51.549493 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.549390    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc044185-a187-4f47-92ee-972bd7794f22-istio-token\" (UniqueName: \"kubernetes.io/projected/bc044185-a187-4f47-92ee-972bd7794f22-istio-token\") pod \"bc044185-a187-4f47-92ee-972bd7794f22\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") "
Oct 31 01:26:51.549493 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.549464    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-workload-certs\") pod \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") "
Oct 31 01:26:51.549667 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.549628    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-workload-certs\") pod \"bc044185-a187-4f47-92ee-972bd7794f22\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.549683    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-workload-socket\") pod \"bc044185-a187-4f47-92ee-972bd7794f22\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.549726    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istiod-ca-cert\") pod \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.549744    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-workload-certs" pod "2a84b045-58f1-43bf-b02b-6ee4c9a62725" (UID: "2a84b045-58f1-43bf-b02b-6ee4c9a62725"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.549774    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc044185-a187-4f47-92ee-972bd7794f22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc044185-a187-4f47-92ee-972bd7794f22-istio-podinfo\") pod \"bc044185-a187-4f47-92ee-972bd7794f22\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.549806    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-credential-socket\") pod \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.549834    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-istio-envoy\") pod \"bc044185-a187-4f47-92ee-972bd7794f22\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.549895    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-workload-socket\") pod \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.549925    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-token\" (UniqueName: \"kubernetes.io/projected/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-token\") pod \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.549948    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-envoy\") pod \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.549998    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-data\") pod \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.550021    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istiod-ca-cert" pod "2a84b045-58f1-43bf-b02b-6ee4c9a62725" (UID: "2a84b045-58f1-43bf-b02b-6ee4c9a62725"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.550031    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-istio-data\") pod \"bc044185-a187-4f47-92ee-972bd7794f22\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.550059    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-credential-socket\") pod \"bc044185-a187-4f47-92ee-972bd7794f22\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.550147    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc044185-a187-4f47-92ee-972bd7794f22-kube-api-access-5vf94\" (UniqueName: \"kubernetes.io/projected/bc044185-a187-4f47-92ee-972bd7794f22-kube-api-access-5vf94\") pod \"bc044185-a187-4f47-92ee-972bd7794f22\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.550222    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-podinfo\") pod \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\" (UID: \"2a84b045-58f1-43bf-b02b-6ee4c9a62725\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.550281    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc044185-a187-4f47-92ee-972bd7794f22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc044185-a187-4f47-92ee-972bd7794f22-istiod-ca-cert\") pod \"bc044185-a187-4f47-92ee-972bd7794f22\" (UID: \"bc044185-a187-4f47-92ee-972bd7794f22\") "
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.550300    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-credential-socket" pod "2a84b045-58f1-43bf-b02b-6ee4c9a62725" (UID: "2a84b045-58f1-43bf-b02b-6ee4c9a62725"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.550501    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-data" pod "2a84b045-58f1-43bf-b02b-6ee4c9a62725" (UID: "2a84b045-58f1-43bf-b02b-6ee4c9a62725"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.550586    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-workload-socket" pod "bc044185-a187-4f47-92ee-972bd7794f22" (UID: "bc044185-a187-4f47-92ee-972bd7794f22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:51.550764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.550677    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-workload-certs" pod "bc044185-a187-4f47-92ee-972bd7794f22" (UID: "bc044185-a187-4f47-92ee-972bd7794f22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:51.551839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.551359    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.551839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.551371    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-istio-data" pod "bc044185-a187-4f47-92ee-972bd7794f22" (UID: "bc044185-a187-4f47-92ee-972bd7794f22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:51.551839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.551381    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.551839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.551397    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.551839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.551413    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.551839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.551428    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.551839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.551442    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.551839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.551396    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-workload-socket" pod "2a84b045-58f1-43bf-b02b-6ee4c9a62725" (UID: "2a84b045-58f1-43bf-b02b-6ee4c9a62725"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:51.551839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.551480    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc044185-a187-4f47-92ee-972bd7794f22-istiod-ca-cert" pod "bc044185-a187-4f47-92ee-972bd7794f22" (UID: "bc044185-a187-4f47-92ee-972bd7794f22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:51.553535 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.553493    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-credential-socket" pod "bc044185-a187-4f47-92ee-972bd7794f22" (UID: "bc044185-a187-4f47-92ee-972bd7794f22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:51.553830 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.553784    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a84b045-58f1-43bf-b02b-6ee4c9a62725-kube-api-access-2vhfp" pod "2a84b045-58f1-43bf-b02b-6ee4c9a62725" (UID: "2a84b045-58f1-43bf-b02b-6ee4c9a62725"). InnerVolumeSpecName "kube-api-access-2vhfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:51.554094 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.554048    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc044185-a187-4f47-92ee-972bd7794f22-istio-token" pod "bc044185-a187-4f47-92ee-972bd7794f22" (UID: "bc044185-a187-4f47-92ee-972bd7794f22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:51.554475 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.554454    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-envoy" pod "2a84b045-58f1-43bf-b02b-6ee4c9a62725" (UID: "2a84b045-58f1-43bf-b02b-6ee4c9a62725"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:51.554837 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.554818    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-istio-envoy" pod "bc044185-a187-4f47-92ee-972bd7794f22" (UID: "bc044185-a187-4f47-92ee-972bd7794f22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:51.555023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.554986    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc044185-a187-4f47-92ee-972bd7794f22-istio-podinfo" pod "bc044185-a187-4f47-92ee-972bd7794f22" (UID: "bc044185-a187-4f47-92ee-972bd7794f22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:26:51.555108 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.555067    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc044185-a187-4f47-92ee-972bd7794f22-kube-api-access-5vf94" pod "bc044185-a187-4f47-92ee-972bd7794f22" (UID: "bc044185-a187-4f47-92ee-972bd7794f22"). InnerVolumeSpecName "kube-api-access-5vf94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:51.556157 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.556130    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-podinfo" pod "2a84b045-58f1-43bf-b02b-6ee4c9a62725" (UID: "2a84b045-58f1-43bf-b02b-6ee4c9a62725"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:26:51.556157 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.556149    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-token" pod "2a84b045-58f1-43bf-b02b-6ee4c9a62725" (UID: "2a84b045-58f1-43bf-b02b-6ee4c9a62725"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:51.652987 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.652954    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.652987 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.652983    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.652987 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.652992    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.653220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.653001    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.653220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.653010    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.653220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.653019    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vf94\" (UniqueName: \"kubernetes.io/projected/bc044185-a187-4f47-92ee-972bd7794f22-kube-api-access-5vf94\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.653220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.653029    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a84b045-58f1-43bf-b02b-6ee4c9a62725-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.653220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.653037    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc044185-a187-4f47-92ee-972bd7794f22-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.653220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.653046    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vhfp\" (UniqueName: \"kubernetes.io/projected/2a84b045-58f1-43bf-b02b-6ee4c9a62725-kube-api-access-2vhfp\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.653220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.653056    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc044185-a187-4f47-92ee-972bd7794f22-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.653220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.653064    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc044185-a187-4f47-92ee-972bd7794f22-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.653220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.653071    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc044185-a187-4f47-92ee-972bd7794f22-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:51.993015 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.992980    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:26:51.993201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:51.993046    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn" podUID="a97eb81c-138f-42d7-a47e-a19504d68e7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:26:52.203363 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.203332    4476 generic.go:356] "Generic (PLEG): container finished" podID="2a84b045-58f1-43bf-b02b-6ee4c9a62725" containerID="bb14bd08eb6fa6ae42de87d7d657a2e25aa5dc27f59777916da8b7367a7200d1" exitCode=0
Oct 31 01:26:52.203517 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.203379    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"
Oct 31 01:26:52.203517 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.203409    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7" event={"ID":"2a84b045-58f1-43bf-b02b-6ee4c9a62725","Type":"ContainerDied","Data":"bb14bd08eb6fa6ae42de87d7d657a2e25aa5dc27f59777916da8b7367a7200d1"}
Oct 31 01:26:52.203517 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.203466    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7" event={"ID":"2a84b045-58f1-43bf-b02b-6ee4c9a62725","Type":"ContainerDied","Data":"4e309f5e1eef760941902736aa8846b42d3134d114db2357ee22f81752fab4d6"}
Oct 31 01:26:52.203517 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.203496    4476 scope.go:122] "RemoveContainer" containerID="bb14bd08eb6fa6ae42de87d7d657a2e25aa5dc27f59777916da8b7367a7200d1"
Oct 31 01:26:52.204966 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.204943    4476 generic.go:356] "Generic (PLEG): container finished" podID="bc044185-a187-4f47-92ee-972bd7794f22" containerID="41355714b0b6e7f323fc30dcce040dbd85d6c85fd06f0fddae8cf6fe71471418" exitCode=0
Oct 31 01:26:52.205087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.204989    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"
Oct 31 01:26:52.205087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.205025    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7" event={"ID":"bc044185-a187-4f47-92ee-972bd7794f22","Type":"ContainerDied","Data":"41355714b0b6e7f323fc30dcce040dbd85d6c85fd06f0fddae8cf6fe71471418"}
Oct 31 01:26:52.205087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.205074    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7" event={"ID":"bc044185-a187-4f47-92ee-972bd7794f22","Type":"ContainerDied","Data":"27bcf987c7198ebc2c4a5b4079513874df6dcfbfe51190c28bc06bb13aa12316"}
Oct 31 01:26:52.213720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.213702    4476 scope.go:122] "RemoveContainer" containerID="bb14bd08eb6fa6ae42de87d7d657a2e25aa5dc27f59777916da8b7367a7200d1"
Oct 31 01:26:52.214095 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:26:52.213944    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb14bd08eb6fa6ae42de87d7d657a2e25aa5dc27f59777916da8b7367a7200d1\": container with ID starting with bb14bd08eb6fa6ae42de87d7d657a2e25aa5dc27f59777916da8b7367a7200d1 not found: ID does not exist" containerID="bb14bd08eb6fa6ae42de87d7d657a2e25aa5dc27f59777916da8b7367a7200d1"
Oct 31 01:26:52.214095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.213971    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb14bd08eb6fa6ae42de87d7d657a2e25aa5dc27f59777916da8b7367a7200d1"} err="failed to get container status \"bb14bd08eb6fa6ae42de87d7d657a2e25aa5dc27f59777916da8b7367a7200d1\": rpc error: code = NotFound desc = could not find container \"bb14bd08eb6fa6ae42de87d7d657a2e25aa5dc27f59777916da8b7367a7200d1\": container with ID starting with bb14bd08eb6fa6ae42de87d7d657a2e25aa5dc27f59777916da8b7367a7200d1 not found: ID does not exist"
Oct 31 01:26:52.214095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.213994    4476 scope.go:122] "RemoveContainer" containerID="41355714b0b6e7f323fc30dcce040dbd85d6c85fd06f0fddae8cf6fe71471418"
Oct 31 01:26:52.224448 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.224433    4476 scope.go:122] "RemoveContainer" containerID="41355714b0b6e7f323fc30dcce040dbd85d6c85fd06f0fddae8cf6fe71471418"
Oct 31 01:26:52.224694 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:26:52.224676    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41355714b0b6e7f323fc30dcce040dbd85d6c85fd06f0fddae8cf6fe71471418\": container with ID starting with 41355714b0b6e7f323fc30dcce040dbd85d6c85fd06f0fddae8cf6fe71471418 not found: ID does not exist" containerID="41355714b0b6e7f323fc30dcce040dbd85d6c85fd06f0fddae8cf6fe71471418"
Oct 31 01:26:52.224761 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.224704    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41355714b0b6e7f323fc30dcce040dbd85d6c85fd06f0fddae8cf6fe71471418"} err="failed to get container status \"41355714b0b6e7f323fc30dcce040dbd85d6c85fd06f0fddae8cf6fe71471418\": rpc error: code = NotFound desc = could not find container \"41355714b0b6e7f323fc30dcce040dbd85d6c85fd06f0fddae8cf6fe71471418\": container with ID starting with 41355714b0b6e7f323fc30dcce040dbd85d6c85fd06f0fddae8cf6fe71471418 not found: ID does not exist"
Oct 31 01:26:52.371458 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.371432    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"]
Oct 31 01:26:52.413550 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.413524    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:26:52.413645 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.413583    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7" podUID="bc044185-a187-4f47-92ee-972bd7794f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:52.421006 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.420984    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-xfgv7"]
Oct 31 01:26:52.456703 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.456678    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c75060f-8290-4054-826e-c0044394f5d5" path="/var/lib/kubelet/pods/1c75060f-8290-4054-826e-c0044394f5d5/volumes"
Oct 31 01:26:52.457214 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.457198    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a84b045-58f1-43bf-b02b-6ee4c9a62725" path="/var/lib/kubelet/pods/2a84b045-58f1-43bf-b02b-6ee4c9a62725/volumes"
Oct 31 01:26:52.491001 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.490979    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"]
Oct 31 01:26:52.497803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.497781    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-c6bh7"]
Oct 31 01:26:52.814047 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.814017    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:26:52.818995 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.818975    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:26:52.927402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.927377    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:26:52.937403 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.937373    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:26:52.940861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.940836    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-data\") pod \"fb63a95e-790b-425c-bb97-9925b8f20a49\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") "
Oct 31 01:26:52.940975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.940881    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-workload-certs\") pod \"fb63a95e-790b-425c-bb97-9925b8f20a49\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") "
Oct 31 01:26:52.940975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.940932    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-envoy\") pod \"fb63a95e-790b-425c-bb97-9925b8f20a49\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") "
Oct 31 01:26:52.940975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.940957    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-podinfo\") pod \"fb63a95e-790b-425c-bb97-9925b8f20a49\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") "
Oct 31 01:26:52.941148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.940998    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-token\" (UniqueName: \"kubernetes.io/projected/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-token\") pod \"fb63a95e-790b-425c-bb97-9925b8f20a49\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") "
Oct 31 01:26:52.941148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941025    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-workload-socket\") pod \"fb63a95e-790b-425c-bb97-9925b8f20a49\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") "
Oct 31 01:26:52.941148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941082    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-credential-socket\") pod \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") "
Oct 31 01:26:52.941148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941126    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-workload-socket\") pod \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") "
Oct 31 01:26:52.941376 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941158    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-data" pod "fb63a95e-790b-425c-bb97-9925b8f20a49" (UID: "fb63a95e-790b-425c-bb97-9925b8f20a49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:52.941376 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941205    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-workload-certs\") pod \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") "
Oct 31 01:26:52.941376 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941216    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-workload-certs" pod "fb63a95e-790b-425c-bb97-9925b8f20a49" (UID: "fb63a95e-790b-425c-bb97-9925b8f20a49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:52.941376 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941265    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c71078d-3e0f-4e16-9df8-7cabafce5af9-kube-api-access-x8tfk\" (UniqueName: \"kubernetes.io/projected/7c71078d-3e0f-4e16-9df8-7cabafce5af9-kube-api-access-x8tfk\") pod \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") "
Oct 31 01:26:52.941376 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941321    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-podinfo\") pod \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") "
Oct 31 01:26:52.941668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941402    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-credential-socket\") pod \"fb63a95e-790b-425c-bb97-9925b8f20a49\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") "
Oct 31 01:26:52.941668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941404    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-credential-socket" pod "7c71078d-3e0f-4e16-9df8-7cabafce5af9" (UID: "7c71078d-3e0f-4e16-9df8-7cabafce5af9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:52.941668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941444    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-envoy\") pod \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") "
Oct 31 01:26:52.941668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941482    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istiod-ca-cert\") pod \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") "
Oct 31 01:26:52.941668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941489    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-workload-certs" pod "7c71078d-3e0f-4e16-9df8-7cabafce5af9" (UID: "7c71078d-3e0f-4e16-9df8-7cabafce5af9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:52.941668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941526    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-data\") pod \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") "
Oct 31 01:26:52.941668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941551    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb63a95e-790b-425c-bb97-9925b8f20a49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb63a95e-790b-425c-bb97-9925b8f20a49-istiod-ca-cert\") pod \"fb63a95e-790b-425c-bb97-9925b8f20a49\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") "
Oct 31 01:26:52.941668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941578    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-token\" (UniqueName: \"kubernetes.io/projected/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-token\") pod \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\" (UID: \"7c71078d-3e0f-4e16-9df8-7cabafce5af9\") "
Oct 31 01:26:52.941668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941608    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb63a95e-790b-425c-bb97-9925b8f20a49-kube-api-access-twpzn\" (UniqueName: \"kubernetes.io/projected/fb63a95e-790b-425c-bb97-9925b8f20a49-kube-api-access-twpzn\") pod \"fb63a95e-790b-425c-bb97-9925b8f20a49\" (UID: \"fb63a95e-790b-425c-bb97-9925b8f20a49\") "
Oct 31 01:26:52.942075 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.941723    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-workload-socket" pod "7c71078d-3e0f-4e16-9df8-7cabafce5af9" (UID: "7c71078d-3e0f-4e16-9df8-7cabafce5af9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:52.942260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.942235    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:52.942383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.942263    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:52.942383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.942280    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:52.942383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.942294    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:52.942383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.942308    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:52.942605 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.942524    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-data" pod "7c71078d-3e0f-4e16-9df8-7cabafce5af9" (UID: "7c71078d-3e0f-4e16-9df8-7cabafce5af9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:52.943066 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.943013    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-credential-socket" pod "fb63a95e-790b-425c-bb97-9925b8f20a49" (UID: "fb63a95e-790b-425c-bb97-9925b8f20a49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:52.943723 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.943694    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-workload-socket" pod "fb63a95e-790b-425c-bb97-9925b8f20a49" (UID: "fb63a95e-790b-425c-bb97-9925b8f20a49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:52.944003 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.943978    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb63a95e-790b-425c-bb97-9925b8f20a49-istiod-ca-cert" pod "fb63a95e-790b-425c-bb97-9925b8f20a49" (UID: "fb63a95e-790b-425c-bb97-9925b8f20a49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:52.944236 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.944213    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c71078d-3e0f-4e16-9df8-7cabafce5af9-kube-api-access-x8tfk" pod "7c71078d-3e0f-4e16-9df8-7cabafce5af9" (UID: "7c71078d-3e0f-4e16-9df8-7cabafce5af9"). InnerVolumeSpecName "kube-api-access-x8tfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:52.944690 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.944669    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istiod-ca-cert" pod "7c71078d-3e0f-4e16-9df8-7cabafce5af9" (UID: "7c71078d-3e0f-4e16-9df8-7cabafce5af9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:52.944986 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.944956    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-envoy" pod "fb63a95e-790b-425c-bb97-9925b8f20a49" (UID: "fb63a95e-790b-425c-bb97-9925b8f20a49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:52.948580 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.948552    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-podinfo" pod "7c71078d-3e0f-4e16-9df8-7cabafce5af9" (UID: "7c71078d-3e0f-4e16-9df8-7cabafce5af9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:26:52.948732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.948710    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-token" pod "fb63a95e-790b-425c-bb97-9925b8f20a49" (UID: "fb63a95e-790b-425c-bb97-9925b8f20a49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:52.953663 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.953641    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-podinfo" pod "fb63a95e-790b-425c-bb97-9925b8f20a49" (UID: "fb63a95e-790b-425c-bb97-9925b8f20a49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:26:52.953764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.953653    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb63a95e-790b-425c-bb97-9925b8f20a49-kube-api-access-twpzn" pod "fb63a95e-790b-425c-bb97-9925b8f20a49" (UID: "fb63a95e-790b-425c-bb97-9925b8f20a49"). InnerVolumeSpecName "kube-api-access-twpzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:52.953764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.953658    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-envoy" pod "7c71078d-3e0f-4e16-9df8-7cabafce5af9" (UID: "7c71078d-3e0f-4e16-9df8-7cabafce5af9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:52.953764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:52.953674    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-token" pod "7c71078d-3e0f-4e16-9df8-7cabafce5af9" (UID: "7c71078d-3e0f-4e16-9df8-7cabafce5af9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:53.044003 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.043855    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-envoy\") pod \"a97eb81c-138f-42d7-a47e-a19504d68e7c\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") "
Oct 31 01:26:53.044462 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.044414    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-workload-certs\") pod \"a97eb81c-138f-42d7-a47e-a19504d68e7c\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") "
Oct 31 01:26:53.044462 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.044452    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-credential-socket\") pod \"689153b0-2780-48a8-b936-7b96ba9ca8b1\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") "
Oct 31 01:26:53.044689 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.044502    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97eb81c-138f-42d7-a47e-a19504d68e7c-kube-api-access-qfmxf\" (UniqueName: \"kubernetes.io/projected/a97eb81c-138f-42d7-a47e-a19504d68e7c-kube-api-access-qfmxf\") pod \"a97eb81c-138f-42d7-a47e-a19504d68e7c\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") "
Oct 31 01:26:53.044689 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.044553    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-podinfo\") pod \"689153b0-2780-48a8-b936-7b96ba9ca8b1\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") "
Oct 31 01:26:53.044689 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.044595    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-credential-socket\") pod \"a97eb81c-138f-42d7-a47e-a19504d68e7c\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") "
Oct 31 01:26:53.044689 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.044631    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/689153b0-2780-48a8-b936-7b96ba9ca8b1-kube-api-access-nkg9g\" (UniqueName: \"kubernetes.io/projected/689153b0-2780-48a8-b936-7b96ba9ca8b1-kube-api-access-nkg9g\") pod \"689153b0-2780-48a8-b936-7b96ba9ca8b1\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") "
Oct 31 01:26:53.044689 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.044680    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-podinfo\") pod \"a97eb81c-138f-42d7-a47e-a19504d68e7c\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") "
Oct 31 01:26:53.045382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.044708    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-workload-certs\") pod \"689153b0-2780-48a8-b936-7b96ba9ca8b1\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") "
Oct 31 01:26:53.045382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.044739    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-envoy\") pod \"689153b0-2780-48a8-b936-7b96ba9ca8b1\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") "
Oct 31 01:26:53.045382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.044799    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a97eb81c-138f-42d7-a47e-a19504d68e7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97eb81c-138f-42d7-a47e-a19504d68e7c-istiod-ca-cert\") pod \"a97eb81c-138f-42d7-a47e-a19504d68e7c\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") "
Oct 31 01:26:53.045382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.044868    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-data\") pod \"a97eb81c-138f-42d7-a47e-a19504d68e7c\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") "
Oct 31 01:26:53.045382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.044901    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-workload-socket\") pod \"a97eb81c-138f-42d7-a47e-a19504d68e7c\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") "
Oct 31 01:26:53.045382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.044935    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/689153b0-2780-48a8-b936-7b96ba9ca8b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689153b0-2780-48a8-b936-7b96ba9ca8b1-istiod-ca-cert\") pod \"689153b0-2780-48a8-b936-7b96ba9ca8b1\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") "
Oct 31 01:26:53.045382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.045034    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-token\" (UniqueName: \"kubernetes.io/projected/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-token\") pod \"689153b0-2780-48a8-b936-7b96ba9ca8b1\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") "
Oct 31 01:26:53.045382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.045066    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-token\" (UniqueName: \"kubernetes.io/projected/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-token\") pod \"a97eb81c-138f-42d7-a47e-a19504d68e7c\" (UID: \"a97eb81c-138f-42d7-a47e-a19504d68e7c\") "
Oct 31 01:26:53.045382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.045118    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-data\") pod \"689153b0-2780-48a8-b936-7b96ba9ca8b1\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") "
Oct 31 01:26:53.046699 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.046615    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-data" pod "a97eb81c-138f-42d7-a47e-a19504d68e7c" (UID: "a97eb81c-138f-42d7-a47e-a19504d68e7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:53.047023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.046995    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-workload-certs" pod "a97eb81c-138f-42d7-a47e-a19504d68e7c" (UID: "a97eb81c-138f-42d7-a47e-a19504d68e7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:53.047222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.047159    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a97eb81c-138f-42d7-a47e-a19504d68e7c-istiod-ca-cert" pod "a97eb81c-138f-42d7-a47e-a19504d68e7c" (UID: "a97eb81c-138f-42d7-a47e-a19504d68e7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:53.047644 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.047553    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-workload-certs" pod "689153b0-2780-48a8-b936-7b96ba9ca8b1" (UID: "689153b0-2780-48a8-b936-7b96ba9ca8b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:53.047644 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.047615    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-workload-socket" pod "a97eb81c-138f-42d7-a47e-a19504d68e7c" (UID: "a97eb81c-138f-42d7-a47e-a19504d68e7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:53.048464 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.047864    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-workload-socket\") pod \"689153b0-2780-48a8-b936-7b96ba9ca8b1\" (UID: \"689153b0-2780-48a8-b936-7b96ba9ca8b1\") "
Oct 31 01:26:53.048464 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.047961    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689153b0-2780-48a8-b936-7b96ba9ca8b1-istiod-ca-cert" pod "689153b0-2780-48a8-b936-7b96ba9ca8b1" (UID: "689153b0-2780-48a8-b936-7b96ba9ca8b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:53.048464 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048362    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-data" pod "689153b0-2780-48a8-b936-7b96ba9ca8b1" (UID: "689153b0-2780-48a8-b936-7b96ba9ca8b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:53.048464 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048427    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-credential-socket" pod "689153b0-2780-48a8-b936-7b96ba9ca8b1" (UID: "689153b0-2780-48a8-b936-7b96ba9ca8b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048583    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-workload-socket" pod "689153b0-2780-48a8-b936-7b96ba9ca8b1" (UID: "689153b0-2780-48a8-b936-7b96ba9ca8b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048669    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-podinfo" pod "689153b0-2780-48a8-b936-7b96ba9ca8b1" (UID: "689153b0-2780-48a8-b936-7b96ba9ca8b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048676    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-envoy" pod "a97eb81c-138f-42d7-a47e-a19504d68e7c" (UID: "a97eb81c-138f-42d7-a47e-a19504d68e7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048807    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twpzn\" (UniqueName: \"kubernetes.io/projected/fb63a95e-790b-425c-bb97-9925b8f20a49-kube-api-access-twpzn\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048832    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048849    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048863    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048876    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048889    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb63a95e-790b-425c-bb97-9925b8f20a49-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048903    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048916    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048930    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048943    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a97eb81c-138f-42d7-a47e-a19504d68e7c-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048958    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048972    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8tfk\" (UniqueName: \"kubernetes.io/projected/7c71078d-3e0f-4e16-9df8-7cabafce5af9-kube-api-access-x8tfk\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.048986    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.049000    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689153b0-2780-48a8-b936-7b96ba9ca8b1-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.049014    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.049027    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.049041    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb63a95e-790b-425c-bb97-9925b8f20a49-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.049054    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.049068    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.049082    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.049096    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.049110    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb63a95e-790b-425c-bb97-9925b8f20a49-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.049123    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.049136    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c71078d-3e0f-4e16-9df8-7cabafce5af9-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.049443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.049295    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689153b0-2780-48a8-b936-7b96ba9ca8b1-kube-api-access-nkg9g" pod "689153b0-2780-48a8-b936-7b96ba9ca8b1" (UID: "689153b0-2780-48a8-b936-7b96ba9ca8b1"). InnerVolumeSpecName "kube-api-access-nkg9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:53.050892 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.050261    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-token" pod "689153b0-2780-48a8-b936-7b96ba9ca8b1" (UID: "689153b0-2780-48a8-b936-7b96ba9ca8b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:53.051139 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.051101    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-credential-socket" pod "a97eb81c-138f-42d7-a47e-a19504d68e7c" (UID: "a97eb81c-138f-42d7-a47e-a19504d68e7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:53.051269 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.051242    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97eb81c-138f-42d7-a47e-a19504d68e7c-kube-api-access-qfmxf" pod "a97eb81c-138f-42d7-a47e-a19504d68e7c" (UID: "a97eb81c-138f-42d7-a47e-a19504d68e7c"). InnerVolumeSpecName "kube-api-access-qfmxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:53.051885 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.051863    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-envoy" pod "689153b0-2780-48a8-b936-7b96ba9ca8b1" (UID: "689153b0-2780-48a8-b936-7b96ba9ca8b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:53.051998 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.051982    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-token" pod "a97eb81c-138f-42d7-a47e-a19504d68e7c" (UID: "a97eb81c-138f-42d7-a47e-a19504d68e7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:53.052482 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.052457    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-podinfo" pod "a97eb81c-138f-42d7-a47e-a19504d68e7c" (UID: "a97eb81c-138f-42d7-a47e-a19504d68e7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:26:53.150415 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.150386    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfmxf\" (UniqueName: \"kubernetes.io/projected/a97eb81c-138f-42d7-a47e-a19504d68e7c-kube-api-access-qfmxf\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.150415 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.150413    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a97eb81c-138f-42d7-a47e-a19504d68e7c-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.150572 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.150422    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkg9g\" (UniqueName: \"kubernetes.io/projected/689153b0-2780-48a8-b936-7b96ba9ca8b1-kube-api-access-nkg9g\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.150572 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.150431    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.150572 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.150440    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.150572 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.150448    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/689153b0-2780-48a8-b936-7b96ba9ca8b1-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.150572 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.150457    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a97eb81c-138f-42d7-a47e-a19504d68e7c-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:53.216362 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.216215    4476 generic.go:356] "Generic (PLEG): container finished" podID="fb63a95e-790b-425c-bb97-9925b8f20a49" containerID="53bfa36de2857038f68263e6b016e60a9e1f3342aaf1fd7038c0f932d31feec0" exitCode=0
Oct 31 01:26:53.216362 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.216251    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7" event={"ID":"fb63a95e-790b-425c-bb97-9925b8f20a49","Type":"ContainerDied","Data":"53bfa36de2857038f68263e6b016e60a9e1f3342aaf1fd7038c0f932d31feec0"}
Oct 31 01:26:53.216362 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.216303    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"
Oct 31 01:26:53.216362 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.216342    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7" event={"ID":"fb63a95e-790b-425c-bb97-9925b8f20a49","Type":"ContainerDied","Data":"585bbcfc441c5fb07d8adaabe5a5361d83a5bafd3b91cc2ad814973b66392daf"}
Oct 31 01:26:53.216681 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.216375    4476 scope.go:122] "RemoveContainer" containerID="53bfa36de2857038f68263e6b016e60a9e1f3342aaf1fd7038c0f932d31feec0"
Oct 31 01:26:53.219072 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.219043    4476 generic.go:356] "Generic (PLEG): container finished" podID="689153b0-2780-48a8-b936-7b96ba9ca8b1" containerID="834a22d5c34b838ae06cd9e04954832bc73bb53e54e8779d13809b00229a07b8" exitCode=0
Oct 31 01:26:53.219230 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.219160    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6" event={"ID":"689153b0-2780-48a8-b936-7b96ba9ca8b1","Type":"ContainerDied","Data":"834a22d5c34b838ae06cd9e04954832bc73bb53e54e8779d13809b00229a07b8"}
Oct 31 01:26:53.219304 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.219252    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6" event={"ID":"689153b0-2780-48a8-b936-7b96ba9ca8b1","Type":"ContainerDied","Data":"d8fa49a4f08fb15a35b744aae40e5d9e243fb93565b980b5b2332daf1cbccaea"}
Oct 31 01:26:53.219394 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.219375    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"
Oct 31 01:26:53.222095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.221737    4476 generic.go:356] "Generic (PLEG): container finished" podID="a97eb81c-138f-42d7-a47e-a19504d68e7c" containerID="512047f077b2b2a616a05c7cea5f61043cddf37c2e03a997234b67031f968ef9" exitCode=0
Oct 31 01:26:53.222095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.221795    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"
Oct 31 01:26:53.222095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.221818    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn" event={"ID":"a97eb81c-138f-42d7-a47e-a19504d68e7c","Type":"ContainerDied","Data":"512047f077b2b2a616a05c7cea5f61043cddf37c2e03a997234b67031f968ef9"}
Oct 31 01:26:53.222095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.221866    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn" event={"ID":"a97eb81c-138f-42d7-a47e-a19504d68e7c","Type":"ContainerDied","Data":"77a310f660e454a676574758c6328162a2ed305e28efe956c0ee9c8fd91a67a0"}
Oct 31 01:26:53.226286 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.226259    4476 generic.go:356] "Generic (PLEG): container finished" podID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerID="041e5e9ea3188212b09d71810fe253fa64f4182925545d8f4caf5bd609a847a9" exitCode=0
Oct 31 01:26:53.226439 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.226421    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"
Oct 31 01:26:53.230373 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.230285    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" event={"ID":"7c71078d-3e0f-4e16-9df8-7cabafce5af9","Type":"ContainerDied","Data":"041e5e9ea3188212b09d71810fe253fa64f4182925545d8f4caf5bd609a847a9"}
Oct 31 01:26:53.230373 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.230338    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv" event={"ID":"7c71078d-3e0f-4e16-9df8-7cabafce5af9","Type":"ContainerDied","Data":"9f6199826e7b3a5306015fb73c803c244730ce109d0fa24eae438cfb576cf5e7"}
Oct 31 01:26:53.231736 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.231605    4476 scope.go:122] "RemoveContainer" containerID="53bfa36de2857038f68263e6b016e60a9e1f3342aaf1fd7038c0f932d31feec0"
Oct 31 01:26:53.232400 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:26:53.232111    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53bfa36de2857038f68263e6b016e60a9e1f3342aaf1fd7038c0f932d31feec0\": container with ID starting with 53bfa36de2857038f68263e6b016e60a9e1f3342aaf1fd7038c0f932d31feec0 not found: ID does not exist" containerID="53bfa36de2857038f68263e6b016e60a9e1f3342aaf1fd7038c0f932d31feec0"
Oct 31 01:26:53.232400 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.232146    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53bfa36de2857038f68263e6b016e60a9e1f3342aaf1fd7038c0f932d31feec0"} err="failed to get container status \"53bfa36de2857038f68263e6b016e60a9e1f3342aaf1fd7038c0f932d31feec0\": rpc error: code = NotFound desc = could not find container \"53bfa36de2857038f68263e6b016e60a9e1f3342aaf1fd7038c0f932d31feec0\": container with ID starting with 53bfa36de2857038f68263e6b016e60a9e1f3342aaf1fd7038c0f932d31feec0 not found: ID does not exist"
Oct 31 01:26:53.232400 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.232198    4476 scope.go:122] "RemoveContainer" containerID="834a22d5c34b838ae06cd9e04954832bc73bb53e54e8779d13809b00229a07b8"
Oct 31 01:26:53.245349 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.245331    4476 scope.go:122] "RemoveContainer" containerID="834a22d5c34b838ae06cd9e04954832bc73bb53e54e8779d13809b00229a07b8"
Oct 31 01:26:53.245638 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:26:53.245612    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834a22d5c34b838ae06cd9e04954832bc73bb53e54e8779d13809b00229a07b8\": container with ID starting with 834a22d5c34b838ae06cd9e04954832bc73bb53e54e8779d13809b00229a07b8 not found: ID does not exist" containerID="834a22d5c34b838ae06cd9e04954832bc73bb53e54e8779d13809b00229a07b8"
Oct 31 01:26:53.245694 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.245649    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834a22d5c34b838ae06cd9e04954832bc73bb53e54e8779d13809b00229a07b8"} err="failed to get container status \"834a22d5c34b838ae06cd9e04954832bc73bb53e54e8779d13809b00229a07b8\": rpc error: code = NotFound desc = could not find container \"834a22d5c34b838ae06cd9e04954832bc73bb53e54e8779d13809b00229a07b8\": container with ID starting with 834a22d5c34b838ae06cd9e04954832bc73bb53e54e8779d13809b00229a07b8 not found: ID does not exist"
Oct 31 01:26:53.245694 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.245675    4476 scope.go:122] "RemoveContainer" containerID="512047f077b2b2a616a05c7cea5f61043cddf37c2e03a997234b67031f968ef9"
Oct 31 01:26:53.254877 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.254857    4476 scope.go:122] "RemoveContainer" containerID="512047f077b2b2a616a05c7cea5f61043cddf37c2e03a997234b67031f968ef9"
Oct 31 01:26:53.255153 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:26:53.255133    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"512047f077b2b2a616a05c7cea5f61043cddf37c2e03a997234b67031f968ef9\": container with ID starting with 512047f077b2b2a616a05c7cea5f61043cddf37c2e03a997234b67031f968ef9 not found: ID does not exist" containerID="512047f077b2b2a616a05c7cea5f61043cddf37c2e03a997234b67031f968ef9"
Oct 31 01:26:53.255259 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.255161    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"512047f077b2b2a616a05c7cea5f61043cddf37c2e03a997234b67031f968ef9"} err="failed to get container status \"512047f077b2b2a616a05c7cea5f61043cddf37c2e03a997234b67031f968ef9\": rpc error: code = NotFound desc = could not find container \"512047f077b2b2a616a05c7cea5f61043cddf37c2e03a997234b67031f968ef9\": container with ID starting with 512047f077b2b2a616a05c7cea5f61043cddf37c2e03a997234b67031f968ef9 not found: ID does not exist"
Oct 31 01:26:53.255259 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.255195    4476 scope.go:122] "RemoveContainer" containerID="041e5e9ea3188212b09d71810fe253fa64f4182925545d8f4caf5bd609a847a9"
Oct 31 01:26:53.264022 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.264005    4476 scope.go:122] "RemoveContainer" containerID="57c12101cd32a0a9e60b93f3e966b3324e78680563b69db77d9ea95543da473f"
Oct 31 01:26:53.274959 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.274933    4476 scope.go:122] "RemoveContainer" containerID="041e5e9ea3188212b09d71810fe253fa64f4182925545d8f4caf5bd609a847a9"
Oct 31 01:26:53.275294 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:26:53.275265    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"041e5e9ea3188212b09d71810fe253fa64f4182925545d8f4caf5bd609a847a9\": container with ID starting with 041e5e9ea3188212b09d71810fe253fa64f4182925545d8f4caf5bd609a847a9 not found: ID does not exist" containerID="041e5e9ea3188212b09d71810fe253fa64f4182925545d8f4caf5bd609a847a9"
Oct 31 01:26:53.275368 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.275299    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"041e5e9ea3188212b09d71810fe253fa64f4182925545d8f4caf5bd609a847a9"} err="failed to get container status \"041e5e9ea3188212b09d71810fe253fa64f4182925545d8f4caf5bd609a847a9\": rpc error: code = NotFound desc = could not find container \"041e5e9ea3188212b09d71810fe253fa64f4182925545d8f4caf5bd609a847a9\": container with ID starting with 041e5e9ea3188212b09d71810fe253fa64f4182925545d8f4caf5bd609a847a9 not found: ID does not exist"
Oct 31 01:26:53.275368 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.275317    4476 scope.go:122] "RemoveContainer" containerID="57c12101cd32a0a9e60b93f3e966b3324e78680563b69db77d9ea95543da473f"
Oct 31 01:26:53.275564 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:26:53.275542    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c12101cd32a0a9e60b93f3e966b3324e78680563b69db77d9ea95543da473f\": container with ID starting with 57c12101cd32a0a9e60b93f3e966b3324e78680563b69db77d9ea95543da473f not found: ID does not exist" containerID="57c12101cd32a0a9e60b93f3e966b3324e78680563b69db77d9ea95543da473f"
Oct 31 01:26:53.275626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.275575    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c12101cd32a0a9e60b93f3e966b3324e78680563b69db77d9ea95543da473f"} err="failed to get container status \"57c12101cd32a0a9e60b93f3e966b3324e78680563b69db77d9ea95543da473f\": rpc error: code = NotFound desc = could not find container \"57c12101cd32a0a9e60b93f3e966b3324e78680563b69db77d9ea95543da473f\": container with ID starting with 57c12101cd32a0a9e60b93f3e966b3324e78680563b69db77d9ea95543da473f not found: ID does not exist"
Oct 31 01:26:53.410901 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.410815    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"]
Oct 31 01:26:53.413933 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.413909    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-qfvmv"]
Oct 31 01:26:53.518609 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.518575    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"]
Oct 31 01:26:53.525007 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.524973    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-bc4n6"]
Oct 31 01:26:53.636492 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.636459    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"]
Oct 31 01:26:53.640219 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.640193    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-jz4l7"]
Oct 31 01:26:53.765458 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.765378    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"]
Oct 31 01:26:53.770863 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.770833    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-vwtvn"]
Oct 31 01:26:53.961265 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:53.961237    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:54.454806 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:54.454778    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="689153b0-2780-48a8-b936-7b96ba9ca8b1" path="/var/lib/kubelet/pods/689153b0-2780-48a8-b936-7b96ba9ca8b1/volumes"
Oct 31 01:26:54.455265 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:54.455244    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" path="/var/lib/kubelet/pods/7c71078d-3e0f-4e16-9df8-7cabafce5af9/volumes"
Oct 31 01:26:54.455710 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:54.455694    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97eb81c-138f-42d7-a47e-a19504d68e7c" path="/var/lib/kubelet/pods/a97eb81c-138f-42d7-a47e-a19504d68e7c/volumes"
Oct 31 01:26:54.456703 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:54.456685    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc044185-a187-4f47-92ee-972bd7794f22" path="/var/lib/kubelet/pods/bc044185-a187-4f47-92ee-972bd7794f22/volumes"
Oct 31 01:26:54.457263 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:54.457247    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb63a95e-790b-425c-bb97-9925b8f20a49" path="/var/lib/kubelet/pods/fb63a95e-790b-425c-bb97-9925b8f20a49/volumes"
Oct 31 01:26:58.424486 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:58.424453    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:58.683891 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:26:58.683813    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:59.181903 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:26:59.181861    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:27:02.991896 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:02.991857    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:04.519533 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:04.519501    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:04.849659 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:04.849624    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:05.051017 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:05.050986    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:05.551495 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:05.551460    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:06.906384 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:06.906344    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:07.122342 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:07.122308    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:07.726751 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:07.726719    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:08.282733 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.282696    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"]
Oct 31 01:27:08.284160 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.284134    4476 cpu_manager.go:414] "containerMap: removing container" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerName="istio-proxy"
Oct 31 01:27:08.284346 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.284331    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerName="istio-proxy"
Oct 31 01:27:08.284462 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.284451    4476 cpu_manager.go:414] "containerMap: removing container" podUID="1c75060f-8290-4054-826e-c0044394f5d5" containerName="istio-proxy"
Oct 31 01:27:08.285789 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285761    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c75060f-8290-4054-826e-c0044394f5d5" containerName="istio-proxy"
Oct 31 01:27:08.285789 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285790    4476 cpu_manager.go:414] "containerMap: removing container" podUID="689153b0-2780-48a8-b936-7b96ba9ca8b1" containerName="istio-proxy"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285798    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="689153b0-2780-48a8-b936-7b96ba9ca8b1" containerName="istio-proxy"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285818    4476 cpu_manager.go:414] "containerMap: removing container" podUID="46f84534-7aab-4133-8f6c-a6b2fa4983fb" containerName="maas-api"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285825    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f84534-7aab-4133-8f6c-a6b2fa4983fb" containerName="maas-api"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285837    4476 cpu_manager.go:414] "containerMap: removing container" podUID="640ee4f4-6d6d-49a8-b2b1-686e39623b85" containerName="payload-processing"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285845    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640ee4f4-6d6d-49a8-b2b1-686e39623b85" containerName="payload-processing"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285873    4476 cpu_manager.go:414] "containerMap: removing container" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerName="istio-proxy"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285880    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerName="istio-proxy"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285897    4476 cpu_manager.go:414] "containerMap: removing container" podUID="bc044185-a187-4f47-92ee-972bd7794f22" containerName="istio-proxy"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285903    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc044185-a187-4f47-92ee-972bd7794f22" containerName="istio-proxy"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285923    4476 cpu_manager.go:414] "containerMap: removing container" podUID="fb63a95e-790b-425c-bb97-9925b8f20a49" containerName="istio-proxy"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285929    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb63a95e-790b-425c-bb97-9925b8f20a49" containerName="istio-proxy"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285947    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2a84b045-58f1-43bf-b02b-6ee4c9a62725" containerName="istio-proxy"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285954    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a84b045-58f1-43bf-b02b-6ee4c9a62725" containerName="istio-proxy"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285967    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5245c2fd-f28f-4965-a210-bc0cf9a10376" containerName="istio-proxy"
Oct 31 01:27:08.285982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285975    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5245c2fd-f28f-4965-a210-bc0cf9a10376" containerName="istio-proxy"
Oct 31 01:27:08.287086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.285995    4476 cpu_manager.go:414] "containerMap: removing container" podUID="a97eb81c-138f-42d7-a47e-a19504d68e7c" containerName="istio-proxy"
Oct 31 01:27:08.287086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.286004    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97eb81c-138f-42d7-a47e-a19504d68e7c" containerName="istio-proxy"
Oct 31 01:27:08.287086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.286156    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97eb81c-138f-42d7-a47e-a19504d68e7c" containerName="istio-proxy"
Oct 31 01:27:08.287086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.286168    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerName="istio-proxy"
Oct 31 01:27:08.287086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.286238    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c75060f-8290-4054-826e-c0044394f5d5" containerName="istio-proxy"
Oct 31 01:27:08.287086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.286248    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="689153b0-2780-48a8-b936-7b96ba9ca8b1" containerName="istio-proxy"
Oct 31 01:27:08.287086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.286259    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="46f84534-7aab-4133-8f6c-a6b2fa4983fb" containerName="maas-api"
Oct 31 01:27:08.287086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.286268    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="640ee4f4-6d6d-49a8-b2b1-686e39623b85" containerName="payload-processing"
Oct 31 01:27:08.287086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.286279    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c71078d-3e0f-4e16-9df8-7cabafce5af9" containerName="istio-proxy"
Oct 31 01:27:08.287086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.286288    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc044185-a187-4f47-92ee-972bd7794f22" containerName="istio-proxy"
Oct 31 01:27:08.287086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.286298    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb63a95e-790b-425c-bb97-9925b8f20a49" containerName="istio-proxy"
Oct 31 01:27:08.287086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.286308    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a84b045-58f1-43bf-b02b-6ee4c9a62725" containerName="istio-proxy"
Oct 31 01:27:08.287086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.286317    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5245c2fd-f28f-4965-a210-bc0cf9a10376" containerName="istio-proxy"
Oct 31 01:27:08.300715 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.300672    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.359731 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.356950    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"]
Oct 31 01:27:08.402119 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.401823    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:08.402119 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.402021    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7870b4-openshift-default-dockercfg-n2tvp\""
Oct 31 01:27:08.463119 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.463086    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-credential-socket\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.463310 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.463137    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-workload-certs\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.463310 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.463268    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-podinfo\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.463402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.463307    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-token\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.463402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.463355    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-data\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.463523 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.463500    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-envoy\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.463627 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.463611    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-workload-socket\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.463744 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.463723    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84kx\" (UniqueName: \"kubernetes.io/projected/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-kube-api-access-l84kx\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.463813 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.463759    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istiod-ca-cert\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.564601 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.564503    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-workload-socket\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.564780 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.564616    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84kx\" (UniqueName: \"kubernetes.io/projected/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-kube-api-access-l84kx\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.564780 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.564641    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istiod-ca-cert\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.564780 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.564704    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-credential-socket\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.564780 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.564741    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-workload-certs\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.565076 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.564799    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-podinfo\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.565076 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.564928    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-workload-socket\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.565076 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.564940    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-token\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.565429 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.565405    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-workload-certs\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.565557 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.565522    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-credential-socket\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.565629 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.565587    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-data\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.565695 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.565678    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-envoy\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.565888 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.565857    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-data\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.566591 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.566571    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istiod-ca-cert\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.568033 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.568009    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-podinfo\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.568033 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.568028    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-envoy\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.622271 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.622244    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-token\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.622446 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.622425    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84kx\" (UniqueName: \"kubernetes.io/projected/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-kube-api-access-l84kx\") pod \"e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") " pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.669769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.669721    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:08.729556 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:08.729504    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:09.086386 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:09.085325    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:09.138948 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:09.138916    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"]
Oct 31 01:27:09.163641 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:09.163599    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:27:09.163772 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:09.163688    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:27:09.163772 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:09.163733    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:27:09.327206 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:09.326999    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:09.374978 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:09.374875    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2" event={"ID":"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6","Type":"ContainerStarted","Data":"877ec2c7ca45181809ec11ff1fcab0c89938cb9e1a71c7a3a37e80fb57f85ea7"}
Oct 31 01:27:09.374978 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:09.374937    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2" event={"ID":"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6","Type":"ContainerStarted","Data":"94694effca29dbbad0fbda645b699535a02797e58e76bacd15f4827d252f9230"}
Oct 31 01:27:09.498785 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:09.498727    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2" podStartSLOduration=1.498713329 podStartE2EDuration="1.498713329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:09.494980157 +0000 UTC m=+1839.184863284" watchObservedRunningTime="2026-10-31 01:27:09.498713329 +0000 UTC m=+1839.188596545"
Oct 31 01:27:09.674256 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:09.672502    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:09.982210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:09.982054    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:10.263019 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:10.262935    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:10.594272 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:10.594233    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:10.676049 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:10.676021    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:11.408206 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:11.407906    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:11.409211 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:11.409168    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:27:11.544482 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:11.544452    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:11.770417 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:11.770341    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:12.384687 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:12.384656    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:12.596260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:12.596229    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:12.817218 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:12.817165    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:15.969002 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:15.968973    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:16.087988 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.087951    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"]
Oct 31 01:27:16.092790 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.092771    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"
Oct 31 01:27:16.163538 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.163502    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f997010-2ff4-4bed-8cac-efae308885a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7\" (UID: \"7f997010-2ff4-4bed-8cac-efae308885a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"
Oct 31 01:27:16.163876 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.163781    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6d2s\" (UniqueName: \"kubernetes.io/projected/7f997010-2ff4-4bed-8cac-efae308885a3-kube-api-access-c6d2s\") pod \"payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7\" (UID: \"7f997010-2ff4-4bed-8cac-efae308885a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"
Oct 31 01:27:16.190008 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.189975    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:16.195764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.195726    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7870b4-dockercfg-sgffp\""
Oct 31 01:27:16.195924 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.195838    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7870b4\""
Oct 31 01:27:16.202257 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.202219    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"]
Oct 31 01:27:16.249453 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.249380    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"]
Oct 31 01:27:16.253465 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.253447    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"
Oct 31 01:27:16.264750 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.264720    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6d2s\" (UniqueName: \"kubernetes.io/projected/7f997010-2ff4-4bed-8cac-efae308885a3-kube-api-access-c6d2s\") pod \"payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7\" (UID: \"7f997010-2ff4-4bed-8cac-efae308885a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"
Oct 31 01:27:16.264904 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.264885    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f997010-2ff4-4bed-8cac-efae308885a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7\" (UID: \"7f997010-2ff4-4bed-8cac-efae308885a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"
Oct 31 01:27:16.265591 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.265571    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f997010-2ff4-4bed-8cac-efae308885a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7\" (UID: \"7f997010-2ff4-4bed-8cac-efae308885a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"
Oct 31 01:27:16.310455 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.303551    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"]
Oct 31 01:27:16.325165 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.325124    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6d2s\" (UniqueName: \"kubernetes.io/projected/7f997010-2ff4-4bed-8cac-efae308885a3-kube-api-access-c6d2s\") pod \"payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7\" (UID: \"7f997010-2ff4-4bed-8cac-efae308885a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"
Oct 31 01:27:16.365881 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.365853    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad11508-ee1b-404c-9b04-93db72bfb27a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw\" (UID: \"5ad11508-ee1b-404c-9b04-93db72bfb27a\") " pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"
Oct 31 01:27:16.366034 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.366000    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbpnz\" (UniqueName: \"kubernetes.io/projected/5ad11508-ee1b-404c-9b04-93db72bfb27a-kube-api-access-jbpnz\") pod \"payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw\" (UID: \"5ad11508-ee1b-404c-9b04-93db72bfb27a\") " pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"
Oct 31 01:27:16.407799 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.407768    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"
Oct 31 01:27:16.467033 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.467003    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbpnz\" (UniqueName: \"kubernetes.io/projected/5ad11508-ee1b-404c-9b04-93db72bfb27a-kube-api-access-jbpnz\") pod \"payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw\" (UID: \"5ad11508-ee1b-404c-9b04-93db72bfb27a\") " pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"
Oct 31 01:27:16.467239 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.467221    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad11508-ee1b-404c-9b04-93db72bfb27a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw\" (UID: \"5ad11508-ee1b-404c-9b04-93db72bfb27a\") " pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"
Oct 31 01:27:16.468391 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.468363    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad11508-ee1b-404c-9b04-93db72bfb27a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw\" (UID: \"5ad11508-ee1b-404c-9b04-93db72bfb27a\") " pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"
Oct 31 01:27:16.528048 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.528022    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbpnz\" (UniqueName: \"kubernetes.io/projected/5ad11508-ee1b-404c-9b04-93db72bfb27a-kube-api-access-jbpnz\") pod \"payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw\" (UID: \"5ad11508-ee1b-404c-9b04-93db72bfb27a\") " pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"
Oct 31 01:27:16.567797 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.567761    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"
Oct 31 01:27:16.789142 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.789118    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"]
Oct 31 01:27:16.790112 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:27:16.790083    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f997010_2ff4_4bed_8cac_efae308885a3.slice/crio-0685c135fcabfd0e22250e403941e1c7a7adfabec193529c30a11d7fc77c7523 WatchSource:0}: Error finding container 0685c135fcabfd0e22250e403941e1c7a7adfabec193529c30a11d7fc77c7523: Status 404 returned error can't find the container with id 0685c135fcabfd0e22250e403941e1c7a7adfabec193529c30a11d7fc77c7523
Oct 31 01:27:16.943428 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:16.943408    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"]
Oct 31 01:27:16.944940 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:27:16.944915    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad11508_ee1b_404c_9b04_93db72bfb27a.slice/crio-a9fb1f1b066cfe8750910db7f693a6e342bf1c2ec0afcbf61c1be46ca146c031 WatchSource:0}: Error finding container a9fb1f1b066cfe8750910db7f693a6e342bf1c2ec0afcbf61c1be46ca146c031: Status 404 returned error can't find the container with id a9fb1f1b066cfe8750910db7f693a6e342bf1c2ec0afcbf61c1be46ca146c031
Oct 31 01:27:17.446879 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:17.446839    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7" event={"ID":"7f997010-2ff4-4bed-8cac-efae308885a3","Type":"ContainerStarted","Data":"e184be56be7199f2b2bf63403793060aea1c51e5add5df6c562adeec1673b506"}
Oct 31 01:27:17.447383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:17.446906    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7" event={"ID":"7f997010-2ff4-4bed-8cac-efae308885a3","Type":"ContainerStarted","Data":"0685c135fcabfd0e22250e403941e1c7a7adfabec193529c30a11d7fc77c7523"}
Oct 31 01:27:17.447383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:17.447049    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"
Oct 31 01:27:17.448720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:17.448698    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw" event={"ID":"5ad11508-ee1b-404c-9b04-93db72bfb27a","Type":"ContainerStarted","Data":"2f8d1cd14e1fbafcf43ea27ac756e9966172efc26eb6a1374837128bb07ec236"}
Oct 31 01:27:17.448853 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:17.448734    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw" event={"ID":"5ad11508-ee1b-404c-9b04-93db72bfb27a","Type":"ContainerStarted","Data":"a9fb1f1b066cfe8750910db7f693a6e342bf1c2ec0afcbf61c1be46ca146c031"}
Oct 31 01:27:17.448853 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:17.448756    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"
Oct 31 01:27:17.567730 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:17.567667    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7" podStartSLOduration=1.567652566 podStartE2EDuration="1.567652566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:17.566040073 +0000 UTC m=+1847.255923200" watchObservedRunningTime="2026-10-31 01:27:17.567652566 +0000 UTC m=+1847.257535737"
Oct 31 01:27:17.688312 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:17.688262    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw" podStartSLOduration=1.688247537 podStartE2EDuration="1.688247537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:17.68703137 +0000 UTC m=+1847.376914497" watchObservedRunningTime="2026-10-31 01:27:17.688247537 +0000 UTC m=+1847.378130696"
Oct 31 01:27:20.824475 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:20.824393    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:22.021592 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:22.021558    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:25.800563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:25.800521    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:26.002307 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:26.002271    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:28.457446 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:28.457414    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"
Oct 31 01:27:28.457839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:28.457496    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"
Oct 31 01:27:29.181634 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:27:29.181599    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:27:30.431895 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:30.431865    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:30.630689 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:30.630653    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:30.910165 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:30.910133    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:35.878898 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:35.878859    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:41.170154 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:41.170124    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:45.627292 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:45.627226    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:45.821324 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:45.821273    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:46.325365 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:46.325337    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:46.537127 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:46.537097    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:46.754169 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:46.747388    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:51.827869 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:51.827831    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:55.123062 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:55.123030    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:56.653288 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:27:56.653260    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:59.181071 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:27:59.181029    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:28:01.482670 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:01.481692    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:05.291314 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:05.291268    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:06.248673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:06.248593    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:11.488202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:11.487084    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:16.696080 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:16.696028    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:19.088326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:19.088292    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:19.289168 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:19.289131    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:21.866779 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:21.866751    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:22.049158 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:22.049123    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:26.095883 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:26.095855    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:27.242475 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:27.242441    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:29.181997 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:28:29.181950    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:28:30.932688 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:30.932656    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:32.337776 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:32.337745    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:37.393071 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:37.393021    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:42.180786 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:42.180664    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:46.293470 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:46.293432    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:47.191212 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:47.189918    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:47.410411 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:47.410369    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:52.052915 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:52.052858    4476 generic.go:356] "Generic (PLEG): container finished" podID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerID="877ec2c7ca45181809ec11ff1fcab0c89938cb9e1a71c7a3a37e80fb57f85ea7" exitCode=0
Oct 31 01:28:52.053586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:52.052932    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2" event={"ID":"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6","Type":"ContainerDied","Data":"877ec2c7ca45181809ec11ff1fcab0c89938cb9e1a71c7a3a37e80fb57f85ea7"}
Oct 31 01:28:52.053586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:52.053328    4476 scope.go:122] "RemoveContainer" containerID="877ec2c7ca45181809ec11ff1fcab0c89938cb9e1a71c7a3a37e80fb57f85ea7"
Oct 31 01:28:52.054816 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:52.054787    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:28:52.054892 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:52.054853    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:28:52.054892 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:52.054886    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:28:52.466350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:52.466094    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:53.095382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:53.095340    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2" event={"ID":"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6","Type":"ContainerStarted","Data":"c6869f7cdf7f88c0607292465de67e2dd34f17e805a1c1860b42b2d006df5d15"}
Oct 31 01:28:53.680199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:53.671505    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:28:53.680507 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:53.680279    4476 patch_prober.go:29] interesting pod/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 01:28:53.680507 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:53.680349    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2" podUID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 01:28:54.684298 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:54.684263    4476 patch_prober.go:29] interesting pod/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 01:28:54.684726 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:54.684325    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2" podUID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 01:28:55.675536 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:55.675500    4476 patch_prober.go:29] interesting pod/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 01:28:55.675817 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:55.675790    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2" podUID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 01:28:56.408220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:56.408167    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:28:56.684811 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:56.684314    4476 patch_prober.go:29] interesting pod/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 01:28:56.684811 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:56.684374    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2" podUID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 01:28:57.683665 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:57.683629    4476 patch_prober.go:29] interesting pod/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 01:28:57.684817 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:57.684786    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2" podUID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 01:28:57.685698 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:57.684644    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:58.675333 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:58.675304    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:28:58.676360 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:28:58.676337    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:28:59.181443 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:28:59.181408    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:29:01.907501 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:01.907448    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:02.104580 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:02.104537    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:02.894751 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:02.894709    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:07.797144 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:07.797090    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:12.812365 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:12.812331    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:16.081439 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:16.081408    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"]
Oct 31 01:29:16.081974 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:16.081657    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" podUID="b433eeb1-84a1-4010-8e15-070ad662bc27" containerName="istio-proxy" containerID="cri-o://a2407d28f49213e9b71b18690430cb0b8b1dbab9b6cd3f24a67936ddb150e217" gracePeriod=30
Oct 31 01:29:17.157885 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:17.157850    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"]
Oct 31 01:29:17.158318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:17.158112    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerName="istio-proxy" containerID="cri-o://65f5dbd709cf85fdc89e1a38ff2155b466d266d30e05a40a5b627309c8074f5e" gracePeriod=30
Oct 31 01:29:18.031433 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:18.031402    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:18.234546 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:18.234517    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:18.929925 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:18.929895    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:20.400602 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:20.400559    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:29:20.400998 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:20.400632    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:29:21.274673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.274626    4476 generic.go:356] "Generic (PLEG): container finished" podID="b433eeb1-84a1-4010-8e15-070ad662bc27" containerID="a2407d28f49213e9b71b18690430cb0b8b1dbab9b6cd3f24a67936ddb150e217" exitCode=0
Oct 31 01:29:21.274837 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.274711    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" event={"ID":"b433eeb1-84a1-4010-8e15-070ad662bc27","Type":"ContainerDied","Data":"a2407d28f49213e9b71b18690430cb0b8b1dbab9b6cd3f24a67936ddb150e217"}
Oct 31 01:29:21.337647 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.337602    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:29:21.419303 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.419226    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b433eeb1-84a1-4010-8e15-070ad662bc27-kube-api-access-b499p\" (UniqueName: \"kubernetes.io/projected/b433eeb1-84a1-4010-8e15-070ad662bc27-kube-api-access-b499p\") pod \"b433eeb1-84a1-4010-8e15-070ad662bc27\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") "
Oct 31 01:29:21.419722 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.419322    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-workload-socket\") pod \"b433eeb1-84a1-4010-8e15-070ad662bc27\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") "
Oct 31 01:29:21.419722 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.419379    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-envoy\") pod \"b433eeb1-84a1-4010-8e15-070ad662bc27\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") "
Oct 31 01:29:21.419722 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.419410    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b433eeb1-84a1-4010-8e15-070ad662bc27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b433eeb1-84a1-4010-8e15-070ad662bc27-istiod-ca-cert\") pod \"b433eeb1-84a1-4010-8e15-070ad662bc27\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") "
Oct 31 01:29:21.419722 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.419456    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-data\") pod \"b433eeb1-84a1-4010-8e15-070ad662bc27\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") "
Oct 31 01:29:21.419722 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.419482    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-podinfo\") pod \"b433eeb1-84a1-4010-8e15-070ad662bc27\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") "
Oct 31 01:29:21.419722 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.419503    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-credential-socket\") pod \"b433eeb1-84a1-4010-8e15-070ad662bc27\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") "
Oct 31 01:29:21.419722 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.419631    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-workload-socket" pod "b433eeb1-84a1-4010-8e15-070ad662bc27" (UID: "b433eeb1-84a1-4010-8e15-070ad662bc27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:29:21.419722 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.419669    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-token\" (UniqueName: \"kubernetes.io/projected/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-token\") pod \"b433eeb1-84a1-4010-8e15-070ad662bc27\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") "
Oct 31 01:29:21.419722 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.419705    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-workload-certs\") pod \"b433eeb1-84a1-4010-8e15-070ad662bc27\" (UID: \"b433eeb1-84a1-4010-8e15-070ad662bc27\") "
Oct 31 01:29:21.420438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.419859    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-credential-socket" pod "b433eeb1-84a1-4010-8e15-070ad662bc27" (UID: "b433eeb1-84a1-4010-8e15-070ad662bc27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:29:21.420438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.419910    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b433eeb1-84a1-4010-8e15-070ad662bc27-istiod-ca-cert" pod "b433eeb1-84a1-4010-8e15-070ad662bc27" (UID: "b433eeb1-84a1-4010-8e15-070ad662bc27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:29:21.420438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.420032    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-data" pod "b433eeb1-84a1-4010-8e15-070ad662bc27" (UID: "b433eeb1-84a1-4010-8e15-070ad662bc27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:29:21.420438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.420041    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-workload-certs" pod "b433eeb1-84a1-4010-8e15-070ad662bc27" (UID: "b433eeb1-84a1-4010-8e15-070ad662bc27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:29:21.420438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.420318    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:21.420438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.420334    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:21.420438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.420342    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b433eeb1-84a1-4010-8e15-070ad662bc27-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:21.420438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.420350    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:21.420438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.420363    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:21.421640 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.421620    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-envoy" pod "b433eeb1-84a1-4010-8e15-070ad662bc27" (UID: "b433eeb1-84a1-4010-8e15-070ad662bc27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:29:21.421903 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.421879    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b433eeb1-84a1-4010-8e15-070ad662bc27-kube-api-access-b499p" pod "b433eeb1-84a1-4010-8e15-070ad662bc27" (UID: "b433eeb1-84a1-4010-8e15-070ad662bc27"). InnerVolumeSpecName "kube-api-access-b499p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:29:21.421988 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.421974    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-podinfo" pod "b433eeb1-84a1-4010-8e15-070ad662bc27" (UID: "b433eeb1-84a1-4010-8e15-070ad662bc27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:29:21.422066 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.422054    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-token" pod "b433eeb1-84a1-4010-8e15-070ad662bc27" (UID: "b433eeb1-84a1-4010-8e15-070ad662bc27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:29:21.441285 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.441262    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:21.521975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.521931    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:21.521975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.521967    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b499p\" (UniqueName: \"kubernetes.io/projected/b433eeb1-84a1-4010-8e15-070ad662bc27-kube-api-access-b499p\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:21.521975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.521983    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:21.522321 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.522000    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b433eeb1-84a1-4010-8e15-070ad662bc27-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:21.621663 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:21.621628    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:22.282346 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.282320    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"
Oct 31 01:29:22.282454 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.282352    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9" event={"ID":"b433eeb1-84a1-4010-8e15-070ad662bc27","Type":"ContainerDied","Data":"df16036aa4494a1650de324e07e425147874a31e3e52fbc908d2587304ada531"}
Oct 31 01:29:22.282454 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.282425    4476 scope.go:122] "RemoveContainer" containerID="a2407d28f49213e9b71b18690430cb0b8b1dbab9b6cd3f24a67936ddb150e217"
Oct 31 01:29:22.284786 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.284762    4476 generic.go:356] "Generic (PLEG): container finished" podID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerID="65f5dbd709cf85fdc89e1a38ff2155b466d266d30e05a40a5b627309c8074f5e" exitCode=0
Oct 31 01:29:22.284901 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.284831    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" event={"ID":"43eb8302-cba6-4f09-a38c-d548c01abb46","Type":"ContainerDied","Data":"65f5dbd709cf85fdc89e1a38ff2155b466d266d30e05a40a5b627309c8074f5e"}
Oct 31 01:29:22.292274 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.292253    4476 scope.go:122] "RemoveContainer" containerID="98b36cc107b9f5365f69d25639d4932f3ec13182c2273fb057ff950bedea398c"
Oct 31 01:29:22.432192 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.432118    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:29:22.501196 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.500997    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"]
Oct 31 01:29:22.533547 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.533512    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-envoy\") pod \"43eb8302-cba6-4f09-a38c-d548c01abb46\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") "
Oct 31 01:29:22.533764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.533641    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-workload-socket\") pod \"43eb8302-cba6-4f09-a38c-d548c01abb46\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") "
Oct 31 01:29:22.533764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.533748    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-credential-socket\") pod \"43eb8302-cba6-4f09-a38c-d548c01abb46\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") "
Oct 31 01:29:22.533892 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.533799    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-data\") pod \"43eb8302-cba6-4f09-a38c-d548c01abb46\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") "
Oct 31 01:29:22.533892 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.533849    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43eb8302-cba6-4f09-a38c-d548c01abb46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43eb8302-cba6-4f09-a38c-d548c01abb46-istiod-ca-cert\") pod \"43eb8302-cba6-4f09-a38c-d548c01abb46\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") "
Oct 31 01:29:22.533892 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.533882    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-workload-certs\") pod \"43eb8302-cba6-4f09-a38c-d548c01abb46\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") "
Oct 31 01:29:22.534046 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.533898    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-workload-socket" pod "43eb8302-cba6-4f09-a38c-d548c01abb46" (UID: "43eb8302-cba6-4f09-a38c-d548c01abb46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:29:22.534046 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.533921    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43eb8302-cba6-4f09-a38c-d548c01abb46-kube-api-access-fr99d\" (UniqueName: \"kubernetes.io/projected/43eb8302-cba6-4f09-a38c-d548c01abb46-kube-api-access-fr99d\") pod \"43eb8302-cba6-4f09-a38c-d548c01abb46\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") "
Oct 31 01:29:22.534046 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.533994    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-podinfo\") pod \"43eb8302-cba6-4f09-a38c-d548c01abb46\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") "
Oct 31 01:29:22.534230 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.534090    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-token\" (UniqueName: \"kubernetes.io/projected/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-token\") pod \"43eb8302-cba6-4f09-a38c-d548c01abb46\" (UID: \"43eb8302-cba6-4f09-a38c-d548c01abb46\") "
Oct 31 01:29:22.534752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.534730    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:22.535327 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.535241    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-data" pod "43eb8302-cba6-4f09-a38c-d548c01abb46" (UID: "43eb8302-cba6-4f09-a38c-d548c01abb46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:29:22.535806 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.535780    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43eb8302-cba6-4f09-a38c-d548c01abb46-istiod-ca-cert" pod "43eb8302-cba6-4f09-a38c-d548c01abb46" (UID: "43eb8302-cba6-4f09-a38c-d548c01abb46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:29:22.536287 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.536256    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-credential-socket" pod "43eb8302-cba6-4f09-a38c-d548c01abb46" (UID: "43eb8302-cba6-4f09-a38c-d548c01abb46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:29:22.536394 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.536287    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-workload-certs" pod "43eb8302-cba6-4f09-a38c-d548c01abb46" (UID: "43eb8302-cba6-4f09-a38c-d548c01abb46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:29:22.537572 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.537524    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-envoy" pod "43eb8302-cba6-4f09-a38c-d548c01abb46" (UID: "43eb8302-cba6-4f09-a38c-d548c01abb46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:29:22.537759 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.537733    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43eb8302-cba6-4f09-a38c-d548c01abb46-kube-api-access-fr99d" pod "43eb8302-cba6-4f09-a38c-d548c01abb46" (UID: "43eb8302-cba6-4f09-a38c-d548c01abb46"). InnerVolumeSpecName "kube-api-access-fr99d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:29:22.537882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.537849    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-podinfo" pod "43eb8302-cba6-4f09-a38c-d548c01abb46" (UID: "43eb8302-cba6-4f09-a38c-d548c01abb46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:29:22.538040 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.538018    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-token" pod "43eb8302-cba6-4f09-a38c-d548c01abb46" (UID: "43eb8302-cba6-4f09-a38c-d548c01abb46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:29:22.548738 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.548714    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-lkgg9"]
Oct 31 01:29:22.636130 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.636093    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:22.636130 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.636133    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:22.636384 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.636149    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43eb8302-cba6-4f09-a38c-d548c01abb46-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:22.636384 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.636165    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:22.636384 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.636198    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr99d\" (UniqueName: \"kubernetes.io/projected/43eb8302-cba6-4f09-a38c-d548c01abb46-kube-api-access-fr99d\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:22.636384 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.636213    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:22.636384 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.636226    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:22.636384 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:22.636239    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43eb8302-cba6-4f09-a38c-d548c01abb46-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:23.042464 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:23.042424    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:23.292313 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:23.292280    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"
Oct 31 01:29:23.292626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:23.292329    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp" event={"ID":"43eb8302-cba6-4f09-a38c-d548c01abb46","Type":"ContainerDied","Data":"4b469c33cad4af8f68e6b7d4bac3c349553e3cd712fc9ea9055ed13769728e4b"}
Oct 31 01:29:23.292626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:23.292392    4476 scope.go:122] "RemoveContainer" containerID="65f5dbd709cf85fdc89e1a38ff2155b466d266d30e05a40a5b627309c8074f5e"
Oct 31 01:29:23.457424 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:23.457393    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"]
Oct 31 01:29:23.506934 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:23.506904    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-sv6mp"]
Oct 31 01:29:24.459077 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:24.459044    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" path="/var/lib/kubelet/pods/43eb8302-cba6-4f09-a38c-d548c01abb46/volumes"
Oct 31 01:29:24.459663 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:24.459645    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b433eeb1-84a1-4010-8e15-070ad662bc27" path="/var/lib/kubelet/pods/b433eeb1-84a1-4010-8e15-070ad662bc27/volumes"
Oct 31 01:29:26.432505 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:26.432474    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:27.786061 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:27.786026    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:29.180984 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:29:29.180944    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:29:30.369086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:30.369035    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:31.275243 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:31.275144    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:32.752365 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:32.752335    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:37.015680 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:37.015634    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:38.571764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:38.571700    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:38.809531 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:38.809468    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:39.019898 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:39.019789    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:43.834839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:43.834803    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:46.614412 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:46.614379    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:48.191504 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:48.191470    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:49.627970 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:49.627931    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:49.947843 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:49.947767    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:50.125744 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:50.125711    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:50.307022 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:50.306990    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:50.789896 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:50.789867    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:53.106530 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:53.106491    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"]
Oct 31 01:29:53.217113 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:53.217083    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"]
Oct 31 01:29:53.515570 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:53.515474    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:53.702110 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:53.702071    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:55.376235 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:55.376202    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:55.805784 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:55.805726    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:55.992386 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:55.992354    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:58.106939 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.106904    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx" podUID="a5f5ee47-434e-438e-8537-7bca802f8658" containerName="payload-processing" containerID="cri-o://c28e7128ced74f0ca38893b4b1811d2f2fc84f7cdc0afd3e913f87443bed9294" gracePeriod=25
Oct 31 01:29:58.217480 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.217397    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v" podUID="4830527e-e546-4295-b5be-6d6cf5aa5129" containerName="payload-pre-processing" containerID="cri-o://8e67fd957893693ca3557edce6af5296cb1a6b945e79bc8774b7df7385c51945" gracePeriod=25
Oct 31 01:29:58.372386 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.372209    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"
Oct 31 01:29:58.450585 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.450554    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f5ee47-434e-438e-8537-7bca802f8658-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f5ee47-434e-438e-8537-7bca802f8658-plugins-config-volume\") pod \"a5f5ee47-434e-438e-8537-7bca802f8658\" (UID: \"a5f5ee47-434e-438e-8537-7bca802f8658\") "
Oct 31 01:29:58.450799 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.450785    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f5ee47-434e-438e-8537-7bca802f8658-kube-api-access-b48vk\" (UniqueName: \"kubernetes.io/projected/a5f5ee47-434e-438e-8537-7bca802f8658-kube-api-access-b48vk\") pod \"a5f5ee47-434e-438e-8537-7bca802f8658\" (UID: \"a5f5ee47-434e-438e-8537-7bca802f8658\") "
Oct 31 01:29:58.450935 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.450912    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f5ee47-434e-438e-8537-7bca802f8658-plugins-config-volume" pod "a5f5ee47-434e-438e-8537-7bca802f8658" (UID: "a5f5ee47-434e-438e-8537-7bca802f8658"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:29:58.451295 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.451278    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f5ee47-434e-438e-8537-7bca802f8658-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:58.452941 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.452919    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f5ee47-434e-438e-8537-7bca802f8658-kube-api-access-b48vk" pod "a5f5ee47-434e-438e-8537-7bca802f8658" (UID: "a5f5ee47-434e-438e-8537-7bca802f8658"). InnerVolumeSpecName "kube-api-access-b48vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:29:58.479276 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.479257    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"
Oct 31 01:29:58.495822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.495796    4476 generic.go:356] "Generic (PLEG): container finished" podID="a5f5ee47-434e-438e-8537-7bca802f8658" containerID="c28e7128ced74f0ca38893b4b1811d2f2fc84f7cdc0afd3e913f87443bed9294" exitCode=0
Oct 31 01:29:58.495926 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.495874    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx" event={"ID":"a5f5ee47-434e-438e-8537-7bca802f8658","Type":"ContainerDied","Data":"c28e7128ced74f0ca38893b4b1811d2f2fc84f7cdc0afd3e913f87443bed9294"}
Oct 31 01:29:58.495926 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.495913    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx" event={"ID":"a5f5ee47-434e-438e-8537-7bca802f8658","Type":"ContainerDied","Data":"062b468f244098a20425dfb1a722817a6e165f962b4c69a7503c9219f68fb578"}
Oct 31 01:29:58.496012 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.495939    4476 scope.go:122] "RemoveContainer" containerID="c28e7128ced74f0ca38893b4b1811d2f2fc84f7cdc0afd3e913f87443bed9294"
Oct 31 01:29:58.496055 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.496044    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"
Oct 31 01:29:58.499560 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.499527    4476 generic.go:356] "Generic (PLEG): container finished" podID="4830527e-e546-4295-b5be-6d6cf5aa5129" containerID="8e67fd957893693ca3557edce6af5296cb1a6b945e79bc8774b7df7385c51945" exitCode=0
Oct 31 01:29:58.499676 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.499600    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v" event={"ID":"4830527e-e546-4295-b5be-6d6cf5aa5129","Type":"ContainerDied","Data":"8e67fd957893693ca3557edce6af5296cb1a6b945e79bc8774b7df7385c51945"}
Oct 31 01:29:58.499676 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.499642    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v" event={"ID":"4830527e-e546-4295-b5be-6d6cf5aa5129","Type":"ContainerDied","Data":"9714823d8cb915f7fa93e835983ae8c7ecef2c9a5faec0b4d10c384450e07c28"}
Oct 31 01:29:58.499676 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.499644    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"
Oct 31 01:29:58.507067 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.507043    4476 scope.go:122] "RemoveContainer" containerID="c28e7128ced74f0ca38893b4b1811d2f2fc84f7cdc0afd3e913f87443bed9294"
Oct 31 01:29:58.507447 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:29:58.507426    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28e7128ced74f0ca38893b4b1811d2f2fc84f7cdc0afd3e913f87443bed9294\": container with ID starting with c28e7128ced74f0ca38893b4b1811d2f2fc84f7cdc0afd3e913f87443bed9294 not found: ID does not exist" containerID="c28e7128ced74f0ca38893b4b1811d2f2fc84f7cdc0afd3e913f87443bed9294"
Oct 31 01:29:58.507536 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.507453    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28e7128ced74f0ca38893b4b1811d2f2fc84f7cdc0afd3e913f87443bed9294"} err="failed to get container status \"c28e7128ced74f0ca38893b4b1811d2f2fc84f7cdc0afd3e913f87443bed9294\": rpc error: code = NotFound desc = could not find container \"c28e7128ced74f0ca38893b4b1811d2f2fc84f7cdc0afd3e913f87443bed9294\": container with ID starting with c28e7128ced74f0ca38893b4b1811d2f2fc84f7cdc0afd3e913f87443bed9294 not found: ID does not exist"
Oct 31 01:29:58.507536 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.507470    4476 scope.go:122] "RemoveContainer" containerID="8e67fd957893693ca3557edce6af5296cb1a6b945e79bc8774b7df7385c51945"
Oct 31 01:29:58.517863 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.517846    4476 scope.go:122] "RemoveContainer" containerID="8e67fd957893693ca3557edce6af5296cb1a6b945e79bc8774b7df7385c51945"
Oct 31 01:29:58.518163 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:29:58.518144    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e67fd957893693ca3557edce6af5296cb1a6b945e79bc8774b7df7385c51945\": container with ID starting with 8e67fd957893693ca3557edce6af5296cb1a6b945e79bc8774b7df7385c51945 not found: ID does not exist" containerID="8e67fd957893693ca3557edce6af5296cb1a6b945e79bc8774b7df7385c51945"
Oct 31 01:29:58.518250 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.518170    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e67fd957893693ca3557edce6af5296cb1a6b945e79bc8774b7df7385c51945"} err="failed to get container status \"8e67fd957893693ca3557edce6af5296cb1a6b945e79bc8774b7df7385c51945\": rpc error: code = NotFound desc = could not find container \"8e67fd957893693ca3557edce6af5296cb1a6b945e79bc8774b7df7385c51945\": container with ID starting with 8e67fd957893693ca3557edce6af5296cb1a6b945e79bc8774b7df7385c51945 not found: ID does not exist"
Oct 31 01:29:58.552490 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.552464    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4830527e-e546-4295-b5be-6d6cf5aa5129-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830527e-e546-4295-b5be-6d6cf5aa5129-plugins-config-volume\") pod \"4830527e-e546-4295-b5be-6d6cf5aa5129\" (UID: \"4830527e-e546-4295-b5be-6d6cf5aa5129\") "
Oct 31 01:29:58.552627 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.552594    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4830527e-e546-4295-b5be-6d6cf5aa5129-kube-api-access-lxz5n\" (UniqueName: \"kubernetes.io/projected/4830527e-e546-4295-b5be-6d6cf5aa5129-kube-api-access-lxz5n\") pod \"4830527e-e546-4295-b5be-6d6cf5aa5129\" (UID: \"4830527e-e546-4295-b5be-6d6cf5aa5129\") "
Oct 31 01:29:58.552913 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.552854    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4830527e-e546-4295-b5be-6d6cf5aa5129-plugins-config-volume" pod "4830527e-e546-4295-b5be-6d6cf5aa5129" (UID: "4830527e-e546-4295-b5be-6d6cf5aa5129"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:29:58.553250 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.553230    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b48vk\" (UniqueName: \"kubernetes.io/projected/a5f5ee47-434e-438e-8537-7bca802f8658-kube-api-access-b48vk\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:58.553340 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.553255    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4830527e-e546-4295-b5be-6d6cf5aa5129-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:58.555214 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.555191    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4830527e-e546-4295-b5be-6d6cf5aa5129-kube-api-access-lxz5n" pod "4830527e-e546-4295-b5be-6d6cf5aa5129" (UID: "4830527e-e546-4295-b5be-6d6cf5aa5129"). InnerVolumeSpecName "kube-api-access-lxz5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:29:58.654230 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.654148    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxz5n\" (UniqueName: \"kubernetes.io/projected/4830527e-e546-4295-b5be-6d6cf5aa5129-kube-api-access-lxz5n\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:58.709906 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.709869    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"]
Oct 31 01:29:58.713513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.713481    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-02c8fd-77c5488457-l9xvx"]
Oct 31 01:29:58.921263 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.921161    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"]
Oct 31 01:29:58.923990 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:29:58.923968    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-02c8fd-7d469c9c9-jqz9v"]
Oct 31 01:29:59.182035 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:29:59.181956    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:30:00.104136 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.104110    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:00.209162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209130    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"]
Oct 31 01:30:00.209726 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209710    4476 cpu_manager.go:414] "containerMap: removing container" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerName="istio-proxy"
Oct 31 01:30:00.209773 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209728    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerName="istio-proxy"
Oct 31 01:30:00.209773 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209760    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b433eeb1-84a1-4010-8e15-070ad662bc27" containerName="istio-proxy"
Oct 31 01:30:00.209773 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209765    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b433eeb1-84a1-4010-8e15-070ad662bc27" containerName="istio-proxy"
Oct 31 01:30:00.209861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209776    4476 cpu_manager.go:414] "containerMap: removing container" podUID="4830527e-e546-4295-b5be-6d6cf5aa5129" containerName="payload-pre-processing"
Oct 31 01:30:00.209861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209781    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4830527e-e546-4295-b5be-6d6cf5aa5129" containerName="payload-pre-processing"
Oct 31 01:30:00.209861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209789    4476 cpu_manager.go:414] "containerMap: removing container" podUID="a5f5ee47-434e-438e-8537-7bca802f8658" containerName="payload-processing"
Oct 31 01:30:00.209861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209794    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f5ee47-434e-438e-8537-7bca802f8658" containerName="payload-processing"
Oct 31 01:30:00.209861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209803    4476 cpu_manager.go:414] "containerMap: removing container" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerName="istio-proxy"
Oct 31 01:30:00.209861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209808    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerName="istio-proxy"
Oct 31 01:30:00.210040 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209878    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerName="istio-proxy"
Oct 31 01:30:00.210040 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209889    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b433eeb1-84a1-4010-8e15-070ad662bc27" containerName="istio-proxy"
Oct 31 01:30:00.210040 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209895    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="4830527e-e546-4295-b5be-6d6cf5aa5129" containerName="payload-pre-processing"
Oct 31 01:30:00.210040 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.209901    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f5ee47-434e-438e-8537-7bca802f8658" containerName="payload-processing"
Oct 31 01:30:00.214501 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.214483    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"]
Oct 31 01:30:00.214627 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.214608    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"
Oct 31 01:30:00.215118 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.215102    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="43eb8302-cba6-4f09-a38c-d548c01abb46" containerName="istio-proxy"
Oct 31 01:30:00.218636 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.218619    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"
Oct 31 01:30:00.250060 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.250037    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"]
Oct 31 01:30:00.253663 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.253649    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"
Oct 31 01:30:00.273239 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.273212    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c979dc7a-14aa-4a28-a1c9-415265b7e48b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9\" (UID: \"c979dc7a-14aa-4a28-a1c9-415265b7e48b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"
Oct 31 01:30:00.273327 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.273250    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7840d177-98d4-4dc7-9b08-18dd2e6aa545-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp\" (UID: \"7840d177-98d4-4dc7-9b08-18dd2e6aa545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"
Oct 31 01:30:00.273371 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.273359    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmg8t\" (UniqueName: \"kubernetes.io/projected/7840d177-98d4-4dc7-9b08-18dd2e6aa545-kube-api-access-cmg8t\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp\" (UID: \"7840d177-98d4-4dc7-9b08-18dd2e6aa545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"
Oct 31 01:30:00.273495 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.273483    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vb8\" (UniqueName: \"kubernetes.io/projected/c979dc7a-14aa-4a28-a1c9-415265b7e48b-kube-api-access-c4vb8\") pod \"maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9\" (UID: \"c979dc7a-14aa-4a28-a1c9-415265b7e48b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"
Oct 31 01:30:00.341127 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.341102    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"]
Oct 31 01:30:00.342815 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.342800    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\""
Oct 31 01:30:00.349729 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.349704    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"]
Oct 31 01:30:00.374534 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.374459    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmg8t\" (UniqueName: \"kubernetes.io/projected/7840d177-98d4-4dc7-9b08-18dd2e6aa545-kube-api-access-cmg8t\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp\" (UID: \"7840d177-98d4-4dc7-9b08-18dd2e6aa545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"
Oct 31 01:30:00.375126 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.374611    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsn7s\" (UniqueName: \"kubernetes.io/projected/32f675bd-de2e-4bc7-b01a-adcacbb50d8f-kube-api-access-fsn7s\") pod \"maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9\" (UID: \"32f675bd-de2e-4bc7-b01a-adcacbb50d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"
Oct 31 01:30:00.375126 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.374667    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32f675bd-de2e-4bc7-b01a-adcacbb50d8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9\" (UID: \"32f675bd-de2e-4bc7-b01a-adcacbb50d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"
Oct 31 01:30:00.375126 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.374724    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vb8\" (UniqueName: \"kubernetes.io/projected/c979dc7a-14aa-4a28-a1c9-415265b7e48b-kube-api-access-c4vb8\") pod \"maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9\" (UID: \"c979dc7a-14aa-4a28-a1c9-415265b7e48b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"
Oct 31 01:30:00.375126 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.374789    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c979dc7a-14aa-4a28-a1c9-415265b7e48b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9\" (UID: \"c979dc7a-14aa-4a28-a1c9-415265b7e48b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"
Oct 31 01:30:00.375126 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.374848    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7840d177-98d4-4dc7-9b08-18dd2e6aa545-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp\" (UID: \"7840d177-98d4-4dc7-9b08-18dd2e6aa545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"
Oct 31 01:30:00.375902 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.375881    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7840d177-98d4-4dc7-9b08-18dd2e6aa545-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp\" (UID: \"7840d177-98d4-4dc7-9b08-18dd2e6aa545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"
Oct 31 01:30:00.376910 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.376888    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c979dc7a-14aa-4a28-a1c9-415265b7e48b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9\" (UID: \"c979dc7a-14aa-4a28-a1c9-415265b7e48b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"
Oct 31 01:30:00.414352 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.414299    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"]
Oct 31 01:30:00.432923 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.432893    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vb8\" (UniqueName: \"kubernetes.io/projected/c979dc7a-14aa-4a28-a1c9-415265b7e48b-kube-api-access-c4vb8\") pod \"maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9\" (UID: \"c979dc7a-14aa-4a28-a1c9-415265b7e48b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"
Oct 31 01:30:00.433528 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.433493    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmg8t\" (UniqueName: \"kubernetes.io/projected/7840d177-98d4-4dc7-9b08-18dd2e6aa545-kube-api-access-cmg8t\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp\" (UID: \"7840d177-98d4-4dc7-9b08-18dd2e6aa545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"
Oct 31 01:30:00.464926 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.464891    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4830527e-e546-4295-b5be-6d6cf5aa5129" path="/var/lib/kubelet/pods/4830527e-e546-4295-b5be-6d6cf5aa5129/volumes"
Oct 31 01:30:00.465480 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.465462    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f5ee47-434e-438e-8537-7bca802f8658" path="/var/lib/kubelet/pods/a5f5ee47-434e-438e-8537-7bca802f8658/volumes"
Oct 31 01:30:00.475905 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.475879    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsn7s\" (UniqueName: \"kubernetes.io/projected/32f675bd-de2e-4bc7-b01a-adcacbb50d8f-kube-api-access-fsn7s\") pod \"maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9\" (UID: \"32f675bd-de2e-4bc7-b01a-adcacbb50d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"
Oct 31 01:30:00.476003 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.475910    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32f675bd-de2e-4bc7-b01a-adcacbb50d8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9\" (UID: \"32f675bd-de2e-4bc7-b01a-adcacbb50d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"
Oct 31 01:30:00.476533 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.476516    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32f675bd-de2e-4bc7-b01a-adcacbb50d8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9\" (UID: \"32f675bd-de2e-4bc7-b01a-adcacbb50d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"
Oct 31 01:30:00.528542 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.528466    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"
Oct 31 01:30:00.532291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.532257    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsn7s\" (UniqueName: \"kubernetes.io/projected/32f675bd-de2e-4bc7-b01a-adcacbb50d8f-kube-api-access-fsn7s\") pod \"maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9\" (UID: \"32f675bd-de2e-4bc7-b01a-adcacbb50d8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"
Oct 31 01:30:00.535644 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.535628    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"
Oct 31 01:30:00.566548 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.566514    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"
Oct 31 01:30:00.922992 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.922966    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"]
Oct 31 01:30:00.923620 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:30:00.923598    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7840d177_98d4_4dc7_9b08_18dd2e6aa545.slice/crio-9ac444b47cd0a1950440fae4ad70f7e3994ff2c36e83519cf772c1da5492e95f WatchSource:0}: Error finding container 9ac444b47cd0a1950440fae4ad70f7e3994ff2c36e83519cf772c1da5492e95f: Status 404 returned error can't find the container with id 9ac444b47cd0a1950440fae4ad70f7e3994ff2c36e83519cf772c1da5492e95f
Oct 31 01:30:00.925468 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.925453    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:30:00.964618 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.964544    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"]
Oct 31 01:30:00.965218 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:30:00.964863    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc979dc7a_14aa_4a28_a1c9_415265b7e48b.slice/crio-ac9d53702df46928b06bf50497750040b87173f6b60325c760bec3a51347c117 WatchSource:0}: Error finding container ac9d53702df46928b06bf50497750040b87173f6b60325c760bec3a51347c117: Status 404 returned error can't find the container with id ac9d53702df46928b06bf50497750040b87173f6b60325c760bec3a51347c117
Oct 31 01:30:00.969206 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:00.968625    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"]
Oct 31 01:30:01.519156 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:01.519125    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9" event={"ID":"32f675bd-de2e-4bc7-b01a-adcacbb50d8f","Type":"ContainerStarted","Data":"597ab777dccb1b833c61d6ea2bdaebf6fe4d2c2fd7bd2109fcff6bb748762a7d"}
Oct 31 01:30:01.520349 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:01.520327    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9" event={"ID":"c979dc7a-14aa-4a28-a1c9-415265b7e48b","Type":"ContainerStarted","Data":"ac9d53702df46928b06bf50497750040b87173f6b60325c760bec3a51347c117"}
Oct 31 01:30:01.521423 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:01.521399    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp" event={"ID":"7840d177-98d4-4dc7-9b08-18dd2e6aa545","Type":"ContainerStarted","Data":"9ac444b47cd0a1950440fae4ad70f7e3994ff2c36e83519cf772c1da5492e95f"}
Oct 31 01:30:04.346495 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:04.346459    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:05.855548 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:05.855514    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:06.089863 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:06.089832    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:06.289604 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:06.289575    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:06.803271 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:06.803237    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:07.571122 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:07.571085    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp" event={"ID":"7840d177-98d4-4dc7-9b08-18dd2e6aa545","Type":"ContainerStarted","Data":"1b074a9e271240d94109b0dd11522504e64a4a5f690909819cd1ac24a0ffb74f"}
Oct 31 01:30:07.684250 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:07.684198    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp" podStartSLOduration=1.621845663 podStartE2EDuration="7.684166321s" totalImagesPullingTime="6.062320658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:07.683004148 +0000 UTC m=+2017.372887273" watchObservedRunningTime="2026-10-31 01:30:07.684166321 +0000 UTC m=+2017.374049447"
Oct 31 01:30:08.578672 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:08.578622    4476 generic.go:356] "Generic (PLEG): container finished" podID="7840d177-98d4-4dc7-9b08-18dd2e6aa545" containerID="1b074a9e271240d94109b0dd11522504e64a4a5f690909819cd1ac24a0ffb74f" exitCode=60
Oct 31 01:30:08.579106 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:08.578724    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp" event={"ID":"7840d177-98d4-4dc7-9b08-18dd2e6aa545","Type":"ContainerDied","Data":"1b074a9e271240d94109b0dd11522504e64a4a5f690909819cd1ac24a0ffb74f"}
Oct 31 01:30:08.579106 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:08.578986    4476 scope.go:122] "RemoveContainer" containerID="1b074a9e271240d94109b0dd11522504e64a4a5f690909819cd1ac24a0ffb74f"
Oct 31 01:30:08.580405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:08.580383    4476 generic.go:356] "Generic (PLEG): container finished" podID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f" containerID="c91d82dc98ef3594bfad47fc2b152549e85fd76793c1225cd730cf4d9d87b907" exitCode=60
Oct 31 01:30:08.580519 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:08.580477    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9" event={"ID":"32f675bd-de2e-4bc7-b01a-adcacbb50d8f","Type":"ContainerDied","Data":"c91d82dc98ef3594bfad47fc2b152549e85fd76793c1225cd730cf4d9d87b907"}
Oct 31 01:30:08.580677 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:08.580660    4476 scope.go:122] "RemoveContainer" containerID="c91d82dc98ef3594bfad47fc2b152549e85fd76793c1225cd730cf4d9d87b907"
Oct 31 01:30:08.582193 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:08.582156    4476 generic.go:356] "Generic (PLEG): container finished" podID="c979dc7a-14aa-4a28-a1c9-415265b7e48b" containerID="69862ac02731a3294c596d93fb64394e8df1106eb1682bcaafeec99874d91d72" exitCode=60
Oct 31 01:30:08.582350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:08.582196    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9" event={"ID":"c979dc7a-14aa-4a28-a1c9-415265b7e48b","Type":"ContainerDied","Data":"69862ac02731a3294c596d93fb64394e8df1106eb1682bcaafeec99874d91d72"}
Oct 31 01:30:08.582439 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:08.582428    4476 scope.go:122] "RemoveContainer" containerID="69862ac02731a3294c596d93fb64394e8df1106eb1682bcaafeec99874d91d72"
Oct 31 01:30:09.589587 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:09.589559    4476 generic.go:356] "Generic (PLEG): container finished" podID="7840d177-98d4-4dc7-9b08-18dd2e6aa545" containerID="076299476bbb1dd13d53d005bf14b2268f5ab4c76dd112b64ade6228ed988412" exitCode=60
Oct 31 01:30:09.589976 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:09.589638    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp" event={"ID":"7840d177-98d4-4dc7-9b08-18dd2e6aa545","Type":"ContainerDied","Data":"076299476bbb1dd13d53d005bf14b2268f5ab4c76dd112b64ade6228ed988412"}
Oct 31 01:30:09.589976 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:09.589689    4476 scope.go:122] "RemoveContainer" containerID="1b074a9e271240d94109b0dd11522504e64a4a5f690909819cd1ac24a0ffb74f"
Oct 31 01:30:09.589976 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:09.589948    4476 scope.go:122] "RemoveContainer" containerID="076299476bbb1dd13d53d005bf14b2268f5ab4c76dd112b64ade6228ed988412"
Oct 31 01:30:09.590330 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:30:09.590302    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp_odh-ai-gateway-infra(7840d177-98d4-4dc7-9b08-18dd2e6aa545)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp" podUID="7840d177-98d4-4dc7-9b08-18dd2e6aa545"
Oct 31 01:30:09.595110 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:09.595089    4476 generic.go:356] "Generic (PLEG): container finished" podID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f" containerID="af5c627c703aff62825a551a8a66fcf10d87d9e7a604aecb95c5b06d019a0e85" exitCode=60
Oct 31 01:30:09.595309 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:09.595212    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9" event={"ID":"32f675bd-de2e-4bc7-b01a-adcacbb50d8f","Type":"ContainerDied","Data":"af5c627c703aff62825a551a8a66fcf10d87d9e7a604aecb95c5b06d019a0e85"}
Oct 31 01:30:09.595522 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:09.595507    4476 scope.go:122] "RemoveContainer" containerID="af5c627c703aff62825a551a8a66fcf10d87d9e7a604aecb95c5b06d019a0e85"
Oct 31 01:30:09.595830 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:30:09.595811    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9_odh-ai-gateway-infra(32f675bd-de2e-4bc7-b01a-adcacbb50d8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9" podUID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f"
Oct 31 01:30:09.597253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:09.597234    4476 generic.go:356] "Generic (PLEG): container finished" podID="c979dc7a-14aa-4a28-a1c9-415265b7e48b" containerID="0a8414d1eae9f10d455e3fe865fcd3114e3c36ec4c0594ec94cb483a6a1033ca" exitCode=60
Oct 31 01:30:09.597355 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:09.597260    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9" event={"ID":"c979dc7a-14aa-4a28-a1c9-415265b7e48b","Type":"ContainerDied","Data":"0a8414d1eae9f10d455e3fe865fcd3114e3c36ec4c0594ec94cb483a6a1033ca"}
Oct 31 01:30:09.597597 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:09.597581    4476 scope.go:122] "RemoveContainer" containerID="0a8414d1eae9f10d455e3fe865fcd3114e3c36ec4c0594ec94cb483a6a1033ca"
Oct 31 01:30:09.597848 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:30:09.597829    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9_odh-ai-gateway-infra(c979dc7a-14aa-4a28-a1c9-415265b7e48b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9" podUID="c979dc7a-14aa-4a28-a1c9-415265b7e48b"
Oct 31 01:30:09.678422 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:09.678402    4476 scope.go:122] "RemoveContainer" containerID="c91d82dc98ef3594bfad47fc2b152549e85fd76793c1225cd730cf4d9d87b907"
Oct 31 01:30:09.691159 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:09.691140    4476 scope.go:122] "RemoveContainer" containerID="69862ac02731a3294c596d93fb64394e8df1106eb1682bcaafeec99874d91d72"
Oct 31 01:30:10.605263 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:10.605228    4476 scope.go:122] "RemoveContainer" containerID="af5c627c703aff62825a551a8a66fcf10d87d9e7a604aecb95c5b06d019a0e85"
Oct 31 01:30:10.605697 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:30:10.605523    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9_odh-ai-gateway-infra(32f675bd-de2e-4bc7-b01a-adcacbb50d8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9" podUID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f"
Oct 31 01:30:10.606859 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:10.606835    4476 scope.go:122] "RemoveContainer" containerID="0a8414d1eae9f10d455e3fe865fcd3114e3c36ec4c0594ec94cb483a6a1033ca"
Oct 31 01:30:10.607098 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:30:10.607081    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9_odh-ai-gateway-infra(c979dc7a-14aa-4a28-a1c9-415265b7e48b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9" podUID="c979dc7a-14aa-4a28-a1c9-415265b7e48b"
Oct 31 01:30:10.608364 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:10.608346    4476 scope.go:122] "RemoveContainer" containerID="076299476bbb1dd13d53d005bf14b2268f5ab4c76dd112b64ade6228ed988412"
Oct 31 01:30:10.608578 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:30:10.608558    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp_odh-ai-gateway-infra(7840d177-98d4-4dc7-9b08-18dd2e6aa545)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp" podUID="7840d177-98d4-4dc7-9b08-18dd2e6aa545"
Oct 31 01:30:11.290033 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:11.289986    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:12.554527 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:12.554490    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:12.746410 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:12.746377    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:13.031056 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:13.031029    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:15.565468 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:15.565431    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:15.601275 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:15.601246    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"]
Oct 31 01:30:15.777960 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:15.777937    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"
Oct 31 01:30:15.807748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:15.807712    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c979dc7a-14aa-4a28-a1c9-415265b7e48b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c979dc7a-14aa-4a28-a1c9-415265b7e48b-maas-api-service-ca\") pod \"c979dc7a-14aa-4a28-a1c9-415265b7e48b\" (UID: \"c979dc7a-14aa-4a28-a1c9-415265b7e48b\") "
Oct 31 01:30:15.807936 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:15.807910    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c979dc7a-14aa-4a28-a1c9-415265b7e48b-kube-api-access-c4vb8\" (UniqueName: \"kubernetes.io/projected/c979dc7a-14aa-4a28-a1c9-415265b7e48b-kube-api-access-c4vb8\") pod \"c979dc7a-14aa-4a28-a1c9-415265b7e48b\" (UID: \"c979dc7a-14aa-4a28-a1c9-415265b7e48b\") "
Oct 31 01:30:15.808100 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:15.808077    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c979dc7a-14aa-4a28-a1c9-415265b7e48b-maas-api-service-ca" pod "c979dc7a-14aa-4a28-a1c9-415265b7e48b" (UID: "c979dc7a-14aa-4a28-a1c9-415265b7e48b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:15.808346 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:15.808328    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c979dc7a-14aa-4a28-a1c9-415265b7e48b-maas-api-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:15.830506 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:15.829608    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c979dc7a-14aa-4a28-a1c9-415265b7e48b-kube-api-access-c4vb8" pod "c979dc7a-14aa-4a28-a1c9-415265b7e48b" (UID: "c979dc7a-14aa-4a28-a1c9-415265b7e48b"). InnerVolumeSpecName "kube-api-access-c4vb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:15.909674 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:15.909638    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4vb8\" (UniqueName: \"kubernetes.io/projected/c979dc7a-14aa-4a28-a1c9-415265b7e48b-kube-api-access-c4vb8\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:16.656026 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:16.655999    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"
Oct 31 01:30:16.656462 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:16.655999    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9" event={"ID":"c979dc7a-14aa-4a28-a1c9-415265b7e48b","Type":"ContainerDied","Data":"ac9d53702df46928b06bf50497750040b87173f6b60325c760bec3a51347c117"}
Oct 31 01:30:16.656462 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:16.656134    4476 scope.go:122] "RemoveContainer" containerID="0a8414d1eae9f10d455e3fe865fcd3114e3c36ec4c0594ec94cb483a6a1033ca"
Oct 31 01:30:16.819415 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:16.819388    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"]
Oct 31 01:30:16.868518 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:16.868486    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-88d9b8-29890170-mg8x9"]
Oct 31 01:30:17.212022 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:17.211942    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:17.485740 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:17.485662    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:17.673161 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:17.673129    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:18.219380 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:18.219350    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:18.454537 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:18.454500    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c979dc7a-14aa-4a28-a1c9-415265b7e48b" path="/var/lib/kubelet/pods/c979dc7a-14aa-4a28-a1c9-415265b7e48b/volumes"
Oct 31 01:30:21.447785 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:21.447752    4476 scope.go:122] "RemoveContainer" containerID="af5c627c703aff62825a551a8a66fcf10d87d9e7a604aecb95c5b06d019a0e85"
Oct 31 01:30:22.589164 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:22.589124    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:22.693091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:22.693008    4476 generic.go:356] "Generic (PLEG): container finished" podID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f" containerID="7e286fb5bfffc8454307b0f705835fd236d80c557ad2b8e3a33468b4f85dcbc3" exitCode=60
Oct 31 01:30:22.693091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:22.693073    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9" event={"ID":"32f675bd-de2e-4bc7-b01a-adcacbb50d8f","Type":"ContainerDied","Data":"7e286fb5bfffc8454307b0f705835fd236d80c557ad2b8e3a33468b4f85dcbc3"}
Oct 31 01:30:22.693296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:22.693156    4476 scope.go:122] "RemoveContainer" containerID="af5c627c703aff62825a551a8a66fcf10d87d9e7a604aecb95c5b06d019a0e85"
Oct 31 01:30:22.693519 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:22.693491    4476 scope.go:122] "RemoveContainer" containerID="7e286fb5bfffc8454307b0f705835fd236d80c557ad2b8e3a33468b4f85dcbc3"
Oct 31 01:30:22.693789 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:30:22.693756    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9_odh-ai-gateway-infra(32f675bd-de2e-4bc7-b01a-adcacbb50d8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9" podUID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f"
Oct 31 01:30:23.817458 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:23.817424    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"]
Oct 31 01:30:23.960612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:23.960591    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"
Oct 31 01:30:23.999947 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:23.999923    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f675bd-de2e-4bc7-b01a-adcacbb50d8f-kube-api-access-fsn7s\" (UniqueName: \"kubernetes.io/projected/32f675bd-de2e-4bc7-b01a-adcacbb50d8f-kube-api-access-fsn7s\") pod \"32f675bd-de2e-4bc7-b01a-adcacbb50d8f\" (UID: \"32f675bd-de2e-4bc7-b01a-adcacbb50d8f\") "
Oct 31 01:30:24.000108 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:24.000002    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f675bd-de2e-4bc7-b01a-adcacbb50d8f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32f675bd-de2e-4bc7-b01a-adcacbb50d8f-maas-api-service-ca\") pod \"32f675bd-de2e-4bc7-b01a-adcacbb50d8f\" (UID: \"32f675bd-de2e-4bc7-b01a-adcacbb50d8f\") "
Oct 31 01:30:24.000364 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:24.000343    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f675bd-de2e-4bc7-b01a-adcacbb50d8f-maas-api-service-ca" pod "32f675bd-de2e-4bc7-b01a-adcacbb50d8f" (UID: "32f675bd-de2e-4bc7-b01a-adcacbb50d8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:24.000546 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:24.000532    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32f675bd-de2e-4bc7-b01a-adcacbb50d8f-maas-api-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:24.002041 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:24.002011    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f675bd-de2e-4bc7-b01a-adcacbb50d8f-kube-api-access-fsn7s" pod "32f675bd-de2e-4bc7-b01a-adcacbb50d8f" (UID: "32f675bd-de2e-4bc7-b01a-adcacbb50d8f"). InnerVolumeSpecName "kube-api-access-fsn7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:24.102105 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:24.102028    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsn7s\" (UniqueName: \"kubernetes.io/projected/32f675bd-de2e-4bc7-b01a-adcacbb50d8f-kube-api-access-fsn7s\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:24.448242 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:24.448153    4476 scope.go:122] "RemoveContainer" containerID="076299476bbb1dd13d53d005bf14b2268f5ab4c76dd112b64ade6228ed988412"
Oct 31 01:30:24.541306 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:24.541281    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:24.707954 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:24.707893    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"
Oct 31 01:30:24.707954 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:24.707895    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9" event={"ID":"32f675bd-de2e-4bc7-b01a-adcacbb50d8f","Type":"ContainerDied","Data":"597ab777dccb1b833c61d6ea2bdaebf6fe4d2c2fd7bd2109fcff6bb748762a7d"}
Oct 31 01:30:24.708141 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:24.707960    4476 scope.go:122] "RemoveContainer" containerID="7e286fb5bfffc8454307b0f705835fd236d80c557ad2b8e3a33468b4f85dcbc3"
Oct 31 01:30:24.728514 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:24.728488    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:24.871918 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:24.871889    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"]
Oct 31 01:30:24.876713 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:24.876683    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-7870b4-29890170-kgvc9"]
Oct 31 01:30:24.924276 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:24.924243    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:25.488684 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:25.488645    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"]
Oct 31 01:30:25.595021 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:25.594977    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"]
Oct 31 01:30:25.717150 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:25.717070    4476 generic.go:356] "Generic (PLEG): container finished" podID="7840d177-98d4-4dc7-9b08-18dd2e6aa545" containerID="3955ef0491227f090871471248353fb5b7f2e91bee651ad6931c8a8d158264f1" exitCode=60
Oct 31 01:30:25.717150 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:25.717140    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp" event={"ID":"7840d177-98d4-4dc7-9b08-18dd2e6aa545","Type":"ContainerDied","Data":"3955ef0491227f090871471248353fb5b7f2e91bee651ad6931c8a8d158264f1"}
Oct 31 01:30:25.717351 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:25.717210    4476 scope.go:122] "RemoveContainer" containerID="076299476bbb1dd13d53d005bf14b2268f5ab4c76dd112b64ade6228ed988412"
Oct 31 01:30:25.717623 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:25.717607    4476 scope.go:122] "RemoveContainer" containerID="3955ef0491227f090871471248353fb5b7f2e91bee651ad6931c8a8d158264f1"
Oct 31 01:30:25.717935 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:30:25.717914    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp_odh-ai-gateway-infra(7840d177-98d4-4dc7-9b08-18dd2e6aa545)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp" podUID="7840d177-98d4-4dc7-9b08-18dd2e6aa545"
Oct 31 01:30:25.880204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:25.880153    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:26.055265 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:26.055230    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:26.454368 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:26.454296    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f" path="/var/lib/kubelet/pods/32f675bd-de2e-4bc7-b01a-adcacbb50d8f/volumes"
Oct 31 01:30:26.762284 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:26.762204    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:26.840692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:26.840662    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"]
Oct 31 01:30:27.078825 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:27.078804    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"
Oct 31 01:30:27.173366 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:27.173336    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7840d177-98d4-4dc7-9b08-18dd2e6aa545-kube-api-access-cmg8t\" (UniqueName: \"kubernetes.io/projected/7840d177-98d4-4dc7-9b08-18dd2e6aa545-kube-api-access-cmg8t\") pod \"7840d177-98d4-4dc7-9b08-18dd2e6aa545\" (UID: \"7840d177-98d4-4dc7-9b08-18dd2e6aa545\") "
Oct 31 01:30:27.173631 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:27.173614    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7840d177-98d4-4dc7-9b08-18dd2e6aa545-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7840d177-98d4-4dc7-9b08-18dd2e6aa545-maas-api-service-ca\") pod \"7840d177-98d4-4dc7-9b08-18dd2e6aa545\" (UID: \"7840d177-98d4-4dc7-9b08-18dd2e6aa545\") "
Oct 31 01:30:27.173932 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:27.173902    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7840d177-98d4-4dc7-9b08-18dd2e6aa545-maas-api-service-ca" pod "7840d177-98d4-4dc7-9b08-18dd2e6aa545" (UID: "7840d177-98d4-4dc7-9b08-18dd2e6aa545"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:27.174127 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:27.174111    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7840d177-98d4-4dc7-9b08-18dd2e6aa545-maas-api-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.175818 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:27.175798    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7840d177-98d4-4dc7-9b08-18dd2e6aa545-kube-api-access-cmg8t" pod "7840d177-98d4-4dc7-9b08-18dd2e6aa545" (UID: "7840d177-98d4-4dc7-9b08-18dd2e6aa545"). InnerVolumeSpecName "kube-api-access-cmg8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:27.274946 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:27.274908    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmg8t\" (UniqueName: \"kubernetes.io/projected/7840d177-98d4-4dc7-9b08-18dd2e6aa545-kube-api-access-cmg8t\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.432226 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:27.432111    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:27.731542 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:27.731457    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp" event={"ID":"7840d177-98d4-4dc7-9b08-18dd2e6aa545","Type":"ContainerDied","Data":"9ac444b47cd0a1950440fae4ad70f7e3994ff2c36e83519cf772c1da5492e95f"}
Oct 31 01:30:27.731542 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:27.731520    4476 scope.go:122] "RemoveContainer" containerID="3955ef0491227f090871471248353fb5b7f2e91bee651ad6931c8a8d158264f1"
Oct 31 01:30:27.731757 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:27.731467    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"
Oct 31 01:30:27.895618 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:27.895585    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"]
Oct 31 01:30:27.899498 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:27.899475    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890170-dj6pp"]
Oct 31 01:30:28.455092 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:28.455056    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7840d177-98d4-4dc7-9b08-18dd2e6aa545" path="/var/lib/kubelet/pods/7840d177-98d4-4dc7-9b08-18dd2e6aa545/volumes"
Oct 31 01:30:29.181649 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:30:29.181613    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:30:30.489939 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.489885    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp" podUID="6ae7c551-8aa2-4728-84ab-c12f662e8387" containerName="payload-processing" containerID="cri-o://6c5e2c85883bae029cef811cf937dac3ef6eb2cc8bb451097fb2fc75f8ade432" gracePeriod=25
Oct 31 01:30:30.595798 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.595752    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l" podUID="e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe" containerName="payload-pre-processing" containerID="cri-o://a634b54093a6a4d69e898bdb3b8b377c4c1bddf2ec3d69dc4760bbd8bbcce04f" gracePeriod=25
Oct 31 01:30:30.678167 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:30:30.678146    4476 logging.go:55] [core] [Channel #2142 SubChannel #2143]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.91:50052", ServerName: "10.133.0.91:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.91:50052: connect: connection refused"
Oct 31 01:30:30.759326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.759298    4476 generic.go:356] "Generic (PLEG): container finished" podID="6ae7c551-8aa2-4728-84ab-c12f662e8387" containerID="6c5e2c85883bae029cef811cf937dac3ef6eb2cc8bb451097fb2fc75f8ade432" exitCode=0
Oct 31 01:30:30.759442 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.759367    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp" event={"ID":"6ae7c551-8aa2-4728-84ab-c12f662e8387","Type":"ContainerDied","Data":"6c5e2c85883bae029cef811cf937dac3ef6eb2cc8bb451097fb2fc75f8ade432"}
Oct 31 01:30:30.759442 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.759417    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp" event={"ID":"6ae7c551-8aa2-4728-84ab-c12f662e8387","Type":"ContainerDied","Data":"89ee120a3192093dbf4bf66c591d26ee8416d633ac48f194ba45017674f840e5"}
Oct 31 01:30:30.759442 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.759439    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89ee120a3192093dbf4bf66c591d26ee8416d633ac48f194ba45017674f840e5"
Oct 31 01:30:30.761091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.761071    4476 generic.go:356] "Generic (PLEG): container finished" podID="e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe" containerID="a634b54093a6a4d69e898bdb3b8b377c4c1bddf2ec3d69dc4760bbd8bbcce04f" exitCode=0
Oct 31 01:30:30.761226 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.761115    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l" event={"ID":"e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe","Type":"ContainerDied","Data":"a634b54093a6a4d69e898bdb3b8b377c4c1bddf2ec3d69dc4760bbd8bbcce04f"}
Oct 31 01:30:30.772475 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.772459    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"
Oct 31 01:30:30.867497 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.867474    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"
Oct 31 01:30:30.916596 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.916565    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae7c551-8aa2-4728-84ab-c12f662e8387-kube-api-access-zd6fm\" (UniqueName: \"kubernetes.io/projected/6ae7c551-8aa2-4728-84ab-c12f662e8387-kube-api-access-zd6fm\") pod \"6ae7c551-8aa2-4728-84ab-c12f662e8387\" (UID: \"6ae7c551-8aa2-4728-84ab-c12f662e8387\") "
Oct 31 01:30:30.916724 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.916677    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae7c551-8aa2-4728-84ab-c12f662e8387-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae7c551-8aa2-4728-84ab-c12f662e8387-plugins-config-volume\") pod \"6ae7c551-8aa2-4728-84ab-c12f662e8387\" (UID: \"6ae7c551-8aa2-4728-84ab-c12f662e8387\") "
Oct 31 01:30:30.917114 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.917085    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae7c551-8aa2-4728-84ab-c12f662e8387-plugins-config-volume" pod "6ae7c551-8aa2-4728-84ab-c12f662e8387" (UID: "6ae7c551-8aa2-4728-84ab-c12f662e8387"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:30.917390 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.917285    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae7c551-8aa2-4728-84ab-c12f662e8387-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.918789 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:30.918766    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae7c551-8aa2-4728-84ab-c12f662e8387-kube-api-access-zd6fm" pod "6ae7c551-8aa2-4728-84ab-c12f662e8387" (UID: "6ae7c551-8aa2-4728-84ab-c12f662e8387"). InnerVolumeSpecName "kube-api-access-zd6fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:31.017904 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:31.017840    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe-plugins-config-volume\") pod \"e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe\" (UID: \"e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe\") "
Oct 31 01:30:31.018036 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:31.018024    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe-kube-api-access-bxsfw\" (UniqueName: \"kubernetes.io/projected/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe-kube-api-access-bxsfw\") pod \"e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe\" (UID: \"e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe\") "
Oct 31 01:30:31.018313 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:31.018282    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe-plugins-config-volume" pod "e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe" (UID: "e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:31.018433 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:31.018349    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd6fm\" (UniqueName: \"kubernetes.io/projected/6ae7c551-8aa2-4728-84ab-c12f662e8387-kube-api-access-zd6fm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:31.020182 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:31.020152    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe-kube-api-access-bxsfw" pod "e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe" (UID: "e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe"). InnerVolumeSpecName "kube-api-access-bxsfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:31.119093 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:31.119070    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxsfw\" (UniqueName: \"kubernetes.io/projected/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe-kube-api-access-bxsfw\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:31.119093 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:31.119092    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:31.678717 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:31.678673    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l" podUID="e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.91:50052\" within 1s: context deadline exceeded"
Oct 31 01:30:31.679085 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:31.678687    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp" podUID="6ae7c551-8aa2-4728-84ab-c12f662e8387" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.92:50052\" within 1s: context deadline exceeded"
Oct 31 01:30:31.679085 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:30:31.678776    4476 logging.go:55] [core] [Channel #2141 SubChannel #2144]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.92:50052", ServerName: "10.133.0.92:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.92:50052: operation was canceled"
Oct 31 01:30:31.768348 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:31.768281    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"
Oct 31 01:30:31.768348 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:31.768294    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l" event={"ID":"e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe","Type":"ContainerDied","Data":"3ca910f31cf79154b059a620fdcef1b270aa83ecb9e1eb77c9cda6726310475f"}
Oct 31 01:30:31.768348 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:31.768337    4476 scope.go:122] "RemoveContainer" containerID="a634b54093a6a4d69e898bdb3b8b377c4c1bddf2ec3d69dc4760bbd8bbcce04f"
Oct 31 01:30:31.768625 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:31.768282    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"
Oct 31 01:30:31.988549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:31.988515    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"]
Oct 31 01:30:32.037987 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:32.037960    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-987f2f-648c6d7d59-n7f8l"]
Oct 31 01:30:32.092039 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:32.092014    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:32.103801 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:32.103779    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"]
Oct 31 01:30:32.106465 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:32.106439    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-987f2f-869dbf49-t5zdp"]
Oct 31 01:30:32.458843 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:32.458759    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae7c551-8aa2-4728-84ab-c12f662e8387" path="/var/lib/kubelet/pods/6ae7c551-8aa2-4728-84ab-c12f662e8387/volumes"
Oct 31 01:30:32.459486 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:32.459463    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe" path="/var/lib/kubelet/pods/e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe/volumes"
Oct 31 01:30:34.084490 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:34.084460    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:37.015221 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:37.015188    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:38.199756 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:38.199719    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"]
Oct 31 01:30:38.200220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:38.199905    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7" podUID="7dc706e1-fe7c-4e88-88a2-ef35706e2791" containerName="istio-proxy" containerID="cri-o://3a5ba4a034839af488cd0969a6f3a5ed18e7ec0f65726c6ee9b8546508447856" gracePeriod=30
Oct 31 01:30:38.868153 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:38.868124    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:39.062735 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:39.062701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:39.243987 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:39.243890    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:41.663122 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:41.663087    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-sxrp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:30:41.663502 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:41.663141    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7" podUID="7dc706e1-fe7c-4e88-88a2-ef35706e2791" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:30:41.902446 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:41.902406    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:43.474043 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.473984    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:30:43.563853 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.563823    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istiod-ca-cert\") pod \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") "
Oct 31 01:30:43.564020 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.563922    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-envoy\") pod \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") "
Oct 31 01:30:43.564020 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.563961    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-podinfo\") pod \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") "
Oct 31 01:30:43.564020 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.564016    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc706e1-fe7c-4e88-88a2-ef35706e2791-kube-api-access-qwbvm\" (UniqueName: \"kubernetes.io/projected/7dc706e1-fe7c-4e88-88a2-ef35706e2791-kube-api-access-qwbvm\") pod \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") "
Oct 31 01:30:43.564147 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.564051    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-workload-socket\") pod \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") "
Oct 31 01:30:43.564147 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.564093    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-data\") pod \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") "
Oct 31 01:30:43.564274 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.564156    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-token\" (UniqueName: \"kubernetes.io/projected/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-token\") pod \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") "
Oct 31 01:30:43.564274 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.564202    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-workload-certs\") pod \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") "
Oct 31 01:30:43.564274 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.564224    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istiod-ca-cert" pod "7dc706e1-fe7c-4e88-88a2-ef35706e2791" (UID: "7dc706e1-fe7c-4e88-88a2-ef35706e2791"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:43.564433 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.564372    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-workload-socket" pod "7dc706e1-fe7c-4e88-88a2-ef35706e2791" (UID: "7dc706e1-fe7c-4e88-88a2-ef35706e2791"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:43.564491 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.564439    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-data" pod "7dc706e1-fe7c-4e88-88a2-ef35706e2791" (UID: "7dc706e1-fe7c-4e88-88a2-ef35706e2791"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:43.564491 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.564461    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-credential-socket\") pod \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\" (UID: \"7dc706e1-fe7c-4e88-88a2-ef35706e2791\") "
Oct 31 01:30:43.564491 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.564464    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-workload-certs" pod "7dc706e1-fe7c-4e88-88a2-ef35706e2791" (UID: "7dc706e1-fe7c-4e88-88a2-ef35706e2791"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:43.564747 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.564720    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-credential-socket" pod "7dc706e1-fe7c-4e88-88a2-ef35706e2791" (UID: "7dc706e1-fe7c-4e88-88a2-ef35706e2791"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:43.565034 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.565002    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:43.565034 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.565025    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:43.565142 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.565039    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:43.565142 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.565053    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:43.565142 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.565066    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:43.566932 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.566904    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-podinfo" pod "7dc706e1-fe7c-4e88-88a2-ef35706e2791" (UID: "7dc706e1-fe7c-4e88-88a2-ef35706e2791"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:30:43.566932 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.566912    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-token" pod "7dc706e1-fe7c-4e88-88a2-ef35706e2791" (UID: "7dc706e1-fe7c-4e88-88a2-ef35706e2791"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:43.567053 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.566987    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-envoy" pod "7dc706e1-fe7c-4e88-88a2-ef35706e2791" (UID: "7dc706e1-fe7c-4e88-88a2-ef35706e2791"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:43.567053 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.567022    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc706e1-fe7c-4e88-88a2-ef35706e2791-kube-api-access-qwbvm" pod "7dc706e1-fe7c-4e88-88a2-ef35706e2791" (UID: "7dc706e1-fe7c-4e88-88a2-ef35706e2791"). InnerVolumeSpecName "kube-api-access-qwbvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:43.666882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.666846    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:43.666882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.666883    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:43.667071 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.666898    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc706e1-fe7c-4e88-88a2-ef35706e2791-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:43.667071 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.666915    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwbvm\" (UniqueName: \"kubernetes.io/projected/7dc706e1-fe7c-4e88-88a2-ef35706e2791-kube-api-access-qwbvm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:43.842973 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.842941    4476 generic.go:356] "Generic (PLEG): container finished" podID="7dc706e1-fe7c-4e88-88a2-ef35706e2791" containerID="3a5ba4a034839af488cd0969a6f3a5ed18e7ec0f65726c6ee9b8546508447856" exitCode=0
Oct 31 01:30:43.843191 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.842986    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"
Oct 31 01:30:43.843191 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.843024    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7" event={"ID":"7dc706e1-fe7c-4e88-88a2-ef35706e2791","Type":"ContainerDied","Data":"3a5ba4a034839af488cd0969a6f3a5ed18e7ec0f65726c6ee9b8546508447856"}
Oct 31 01:30:43.843191 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.843074    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7" event={"ID":"7dc706e1-fe7c-4e88-88a2-ef35706e2791","Type":"ContainerDied","Data":"3c3a7501317f5e6e5831190d48a1aefa322d24148b5c86d92ea09e740213e0b9"}
Oct 31 01:30:43.843191 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.843098    4476 scope.go:122] "RemoveContainer" containerID="3a5ba4a034839af488cd0969a6f3a5ed18e7ec0f65726c6ee9b8546508447856"
Oct 31 01:30:43.856213 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.854007    4476 scope.go:122] "RemoveContainer" containerID="3a5ba4a034839af488cd0969a6f3a5ed18e7ec0f65726c6ee9b8546508447856"
Oct 31 01:30:43.856213 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:30:43.854345    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5ba4a034839af488cd0969a6f3a5ed18e7ec0f65726c6ee9b8546508447856\": container with ID starting with 3a5ba4a034839af488cd0969a6f3a5ed18e7ec0f65726c6ee9b8546508447856 not found: ID does not exist" containerID="3a5ba4a034839af488cd0969a6f3a5ed18e7ec0f65726c6ee9b8546508447856"
Oct 31 01:30:43.856213 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:43.854373    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5ba4a034839af488cd0969a6f3a5ed18e7ec0f65726c6ee9b8546508447856"} err="failed to get container status \"3a5ba4a034839af488cd0969a6f3a5ed18e7ec0f65726c6ee9b8546508447856\": rpc error: code = NotFound desc = could not find container \"3a5ba4a034839af488cd0969a6f3a5ed18e7ec0f65726c6ee9b8546508447856\": container with ID starting with 3a5ba4a034839af488cd0969a6f3a5ed18e7ec0f65726c6ee9b8546508447856 not found: ID does not exist"
Oct 31 01:30:44.006034 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:44.006004    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"]
Oct 31 01:30:44.012366 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:44.012333    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-sxrp7"]
Oct 31 01:30:44.454253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:44.454168    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc706e1-fe7c-4e88-88a2-ef35706e2791" path="/var/lib/kubelet/pods/7dc706e1-fe7c-4e88-88a2-ef35706e2791/volumes"
Oct 31 01:30:44.913600 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:44.913571    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:45.084001 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:45.083971    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:45.965351 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:45.965319    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:46.957672 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:46.957637    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:47.409000 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:47.408954    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:47.590578 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:47.590546    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:47.763498 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:47.763419    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:48.337534 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:48.337494    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:49.635324 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:49.635295    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:49.865972 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:49.865901    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:52.765219 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:52.765128    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:54.070875 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:54.070721    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:54.269337 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:54.269306    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:54.479886 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:54.479795    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:57.693956 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:30:57.693922    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:59.181883 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:30:59.181842    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:31:00.564230 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:00.564164    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:00.898549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:00.898464    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:02.983099 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:02.983062    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:06.650002 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:06.649973    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"]
Oct 31 01:31:06.756022 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:06.755984    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"]
Oct 31 01:31:07.055029 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:07.054975    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:07.246859 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:07.246832    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:07.439624 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:07.439542    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:08.776950 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:08.776913    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:08.992642 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:08.992605    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:09.191113 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:09.191079    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:09.710964 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:09.710932    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:11.650633 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:11.650571    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw" podUID="5ad11508-ee1b-404c-9b04-93db72bfb27a" containerName="payload-processing" containerID="cri-o://2f8d1cd14e1fbafcf43ea27ac756e9966172efc26eb6a1374837128bb07ec236" gracePeriod=25
Oct 31 01:31:11.756589 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:11.756546    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7" podUID="7f997010-2ff4-4bed-8cac-efae308885a3" containerName="payload-pre-processing" containerID="cri-o://e184be56be7199f2b2bf63403793060aea1c51e5add5df6c562adeec1673b506" gracePeriod=25
Oct 31 01:31:11.926863 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:11.926830    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"
Oct 31 01:31:12.010968 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.010920    4476 generic.go:356] "Generic (PLEG): container finished" podID="5ad11508-ee1b-404c-9b04-93db72bfb27a" containerID="2f8d1cd14e1fbafcf43ea27ac756e9966172efc26eb6a1374837128bb07ec236" exitCode=0
Oct 31 01:31:12.011207 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.011124    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw" event={"ID":"5ad11508-ee1b-404c-9b04-93db72bfb27a","Type":"ContainerDied","Data":"2f8d1cd14e1fbafcf43ea27ac756e9966172efc26eb6a1374837128bb07ec236"}
Oct 31 01:31:12.011288 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.011245    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw" event={"ID":"5ad11508-ee1b-404c-9b04-93db72bfb27a","Type":"ContainerDied","Data":"a9fb1f1b066cfe8750910db7f693a6e342bf1c2ec0afcbf61c1be46ca146c031"}
Oct 31 01:31:12.011343 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.011288    4476 scope.go:122] "RemoveContainer" containerID="2f8d1cd14e1fbafcf43ea27ac756e9966172efc26eb6a1374837128bb07ec236"
Oct 31 01:31:12.011509 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.011493    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"
Oct 31 01:31:12.015678 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.015648    4476 generic.go:356] "Generic (PLEG): container finished" podID="7f997010-2ff4-4bed-8cac-efae308885a3" containerID="e184be56be7199f2b2bf63403793060aea1c51e5add5df6c562adeec1673b506" exitCode=0
Oct 31 01:31:12.015813 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.015721    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7" event={"ID":"7f997010-2ff4-4bed-8cac-efae308885a3","Type":"ContainerDied","Data":"e184be56be7199f2b2bf63403793060aea1c51e5add5df6c562adeec1673b506"}
Oct 31 01:31:12.036522 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.036495    4476 scope.go:122] "RemoveContainer" containerID="2f8d1cd14e1fbafcf43ea27ac756e9966172efc26eb6a1374837128bb07ec236"
Oct 31 01:31:12.036870 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:31:12.036843    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8d1cd14e1fbafcf43ea27ac756e9966172efc26eb6a1374837128bb07ec236\": container with ID starting with 2f8d1cd14e1fbafcf43ea27ac756e9966172efc26eb6a1374837128bb07ec236 not found: ID does not exist" containerID="2f8d1cd14e1fbafcf43ea27ac756e9966172efc26eb6a1374837128bb07ec236"
Oct 31 01:31:12.036952 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.036886    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8d1cd14e1fbafcf43ea27ac756e9966172efc26eb6a1374837128bb07ec236"} err="failed to get container status \"2f8d1cd14e1fbafcf43ea27ac756e9966172efc26eb6a1374837128bb07ec236\": rpc error: code = NotFound desc = could not find container \"2f8d1cd14e1fbafcf43ea27ac756e9966172efc26eb6a1374837128bb07ec236\": container with ID starting with 2f8d1cd14e1fbafcf43ea27ac756e9966172efc26eb6a1374837128bb07ec236 not found: ID does not exist"
Oct 31 01:31:12.042317 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.042286    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"
Oct 31 01:31:12.058214 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.058156    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad11508-ee1b-404c-9b04-93db72bfb27a-kube-api-access-jbpnz\" (UniqueName: \"kubernetes.io/projected/5ad11508-ee1b-404c-9b04-93db72bfb27a-kube-api-access-jbpnz\") pod \"5ad11508-ee1b-404c-9b04-93db72bfb27a\" (UID: \"5ad11508-ee1b-404c-9b04-93db72bfb27a\") "
Oct 31 01:31:12.058410 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.058394    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad11508-ee1b-404c-9b04-93db72bfb27a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad11508-ee1b-404c-9b04-93db72bfb27a-plugins-config-volume\") pod \"5ad11508-ee1b-404c-9b04-93db72bfb27a\" (UID: \"5ad11508-ee1b-404c-9b04-93db72bfb27a\") "
Oct 31 01:31:12.058738 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.058714    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad11508-ee1b-404c-9b04-93db72bfb27a-plugins-config-volume" pod "5ad11508-ee1b-404c-9b04-93db72bfb27a" (UID: "5ad11508-ee1b-404c-9b04-93db72bfb27a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:12.058875 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.058857    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad11508-ee1b-404c-9b04-93db72bfb27a-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:12.060439 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.060413    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad11508-ee1b-404c-9b04-93db72bfb27a-kube-api-access-jbpnz" pod "5ad11508-ee1b-404c-9b04-93db72bfb27a" (UID: "5ad11508-ee1b-404c-9b04-93db72bfb27a"). InnerVolumeSpecName "kube-api-access-jbpnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:12.159947 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.159853    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f997010-2ff4-4bed-8cac-efae308885a3-kube-api-access-c6d2s\" (UniqueName: \"kubernetes.io/projected/7f997010-2ff4-4bed-8cac-efae308885a3-kube-api-access-c6d2s\") pod \"7f997010-2ff4-4bed-8cac-efae308885a3\" (UID: \"7f997010-2ff4-4bed-8cac-efae308885a3\") "
Oct 31 01:31:12.160124 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.160094    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f997010-2ff4-4bed-8cac-efae308885a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f997010-2ff4-4bed-8cac-efae308885a3-plugins-config-volume\") pod \"7f997010-2ff4-4bed-8cac-efae308885a3\" (UID: \"7f997010-2ff4-4bed-8cac-efae308885a3\") "
Oct 31 01:31:12.160474 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.160451    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f997010-2ff4-4bed-8cac-efae308885a3-plugins-config-volume" pod "7f997010-2ff4-4bed-8cac-efae308885a3" (UID: "7f997010-2ff4-4bed-8cac-efae308885a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:12.160549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.160469    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbpnz\" (UniqueName: \"kubernetes.io/projected/5ad11508-ee1b-404c-9b04-93db72bfb27a-kube-api-access-jbpnz\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:12.162232 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.162210    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f997010-2ff4-4bed-8cac-efae308885a3-kube-api-access-c6d2s" pod "7f997010-2ff4-4bed-8cac-efae308885a3" (UID: "7f997010-2ff4-4bed-8cac-efae308885a3"). InnerVolumeSpecName "kube-api-access-c6d2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:12.261801 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.261762    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f997010-2ff4-4bed-8cac-efae308885a3-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:12.261801 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.261795    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6d2s\" (UniqueName: \"kubernetes.io/projected/7f997010-2ff4-4bed-8cac-efae308885a3-kube-api-access-c6d2s\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:12.432602 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.432517    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"]
Oct 31 01:31:12.437436 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.437407    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7870b4-5597b7945c-tj6cw"]
Oct 31 01:31:12.454668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:12.454635    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad11508-ee1b-404c-9b04-93db72bfb27a" path="/var/lib/kubelet/pods/5ad11508-ee1b-404c-9b04-93db72bfb27a/volumes"
Oct 31 01:31:13.023224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:13.023163    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7" event={"ID":"7f997010-2ff4-4bed-8cac-efae308885a3","Type":"ContainerDied","Data":"0685c135fcabfd0e22250e403941e1c7a7adfabec193529c30a11d7fc77c7523"}
Oct 31 01:31:13.023224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:13.023220    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"
Oct 31 01:31:13.023739 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:13.023258    4476 scope.go:122] "RemoveContainer" containerID="e184be56be7199f2b2bf63403793060aea1c51e5add5df6c562adeec1673b506"
Oct 31 01:31:13.185244 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:13.185215    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"]
Oct 31 01:31:13.234712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:13.234669    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7870b4-7567f9cd47-rxsg7"]
Oct 31 01:31:14.073166 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:14.072786    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:14.454024 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:14.453943    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f997010-2ff4-4bed-8cac-efae308885a3" path="/var/lib/kubelet/pods/7f997010-2ff4-4bed-8cac-efae308885a3/volumes"
Oct 31 01:31:15.488900 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:15.488870    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:15.674276 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:15.674247    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:15.861763 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:15.861731    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:16.852394 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:16.852364    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:17.727090 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:17.727058    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:17.965079 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:17.965039    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:18.771800 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:18.771762    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:22.156674 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:22.156641    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:22.345584 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:22.345550    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:23.678904 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:23.678867    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:25.334473 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:25.334440    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:25.517737 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:25.517701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:27.107711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:27.107681    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:27.292265 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:27.292232    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:27.993877 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:27.993836    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:28.226547 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:28.226513    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:28.562963 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:28.562930    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:28.872738 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:28.872654    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:29.182253 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:31:29.182153    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:31:29.266311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:29.266277    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:29.463299 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:29.463214    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:30.694004 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:30.693976    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:31:30.718934 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:30.718910    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:31:32.241687 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:32.241646    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:32.443529 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:32.443489    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:32.627909 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:32.627876    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:33.407428 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:33.407392    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:35.170148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:35.170108    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:35.377920 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:35.377890    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:35.576741 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:35.576707    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:35.814402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:35.814365    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:36.089822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:36.089786    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:36.627630 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:36.627588    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:40.395585 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:40.395547    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:41.551293 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:41.551248    4476 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal" podUID="7fc072a2ae8e85ec3464a94d2bac4401" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 31 01:31:41.902447 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:41.902370    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:42.189992 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:42.189915    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:42.377930 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:42.377900    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:43.570680 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:43.570645    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:44.191502 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:44.191447    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:44.460881 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:44.460793    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:45.199302 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:45.199244    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:47.314025 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:47.313989    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:48.374438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:48.374404    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:48.579590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:48.579559    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:48.761060 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:48.760982    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:50.219935 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:50.219898    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:51.615958 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:51.615928    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:51.789906 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:51.789876    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:53.953721 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:53.953685    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:54.138021 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:54.137979    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:54.850204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:54.845395    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:55.037863 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:55.037835    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:55.213161 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:55.213082    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:55.818235 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:55.818192    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:59.182048 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:31:59.182013    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:31:59.662947 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:31:59.662913    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:03.873120 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:03.873087    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:05.365268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:05.365223    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:05.567682 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:05.567646    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:05.743762 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:05.743679    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:06.399783 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:06.399753    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:10.667763 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:10.667734    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:10.857688 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:10.857659    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:11.037805 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:11.037775    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:11.227818 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:11.227788    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:12.196803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:12.196769    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:12.388662 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:12.388627    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:12.497092 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:12.496999    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"]
Oct 31 01:32:12.497284 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:12.497261    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2" podUID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerName="istio-proxy" containerID="cri-o://c6869f7cdf7f88c0607292465de67e2dd34f17e805a1c1860b42b2d006df5d15" gracePeriod=30
Oct 31 01:32:12.568061 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:12.568034    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:13.677921 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:13.677887    4476 patch_prober.go:29] interesting pod/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:32:13.678353 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:13.677955    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2" podUID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:32:15.581064 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:15.581033    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:17.758090 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.758030    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:32:17.860213 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.860156    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-workload-certs\") pod \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") "
Oct 31 01:32:17.860388 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.860246    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-envoy\") pod \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") "
Oct 31 01:32:17.860388 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.860268    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-workload-socket\") pod \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") "
Oct 31 01:32:17.860388 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.860296    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-podinfo\") pod \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") "
Oct 31 01:32:17.860388 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.860328    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-data\") pod \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") "
Oct 31 01:32:17.860615 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.860426    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-workload-certs" pod "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" (UID: "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:17.860615 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.860466    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-credential-socket\") pod \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") "
Oct 31 01:32:17.860615 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.860536    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istiod-ca-cert\") pod \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") "
Oct 31 01:32:17.860615 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.860571    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-kube-api-access-l84kx\" (UniqueName: \"kubernetes.io/projected/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-kube-api-access-l84kx\") pod \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") "
Oct 31 01:32:17.860615 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.860581    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-workload-socket" pod "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" (UID: "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:17.860866 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.860649    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-token\" (UniqueName: \"kubernetes.io/projected/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-token\") pod \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\" (UID: \"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6\") "
Oct 31 01:32:17.860866 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.860638    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-data" pod "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" (UID: "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:17.860866 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.860716    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-credential-socket" pod "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" (UID: "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:17.860866 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.860858    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istiod-ca-cert" pod "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" (UID: "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:32:17.861268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.861210    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:17.861268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.861234    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:17.861268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.861251    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:17.861268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.861265    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:17.861497 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.861279    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:17.862720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.862693    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-envoy" pod "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" (UID: "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:17.862840 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.862728    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-podinfo" pod "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" (UID: "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:32:17.862840 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.862768    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-kube-api-access-l84kx" pod "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" (UID: "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6"). InnerVolumeSpecName "kube-api-access-l84kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:17.862921 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.862883    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-token" pod "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" (UID: "b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:17.962440 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.962397    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:17.962440 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.962429    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:17.962440 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.962440    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l84kx\" (UniqueName: \"kubernetes.io/projected/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-kube-api-access-l84kx\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:17.962440 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:17.962449    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:18.431351 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:18.431318    4476 generic.go:356] "Generic (PLEG): container finished" podID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerID="c6869f7cdf7f88c0607292465de67e2dd34f17e805a1c1860b42b2d006df5d15" exitCode=0
Oct 31 01:32:18.431535 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:18.431372    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"
Oct 31 01:32:18.431535 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:18.431407    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2" event={"ID":"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6","Type":"ContainerDied","Data":"c6869f7cdf7f88c0607292465de67e2dd34f17e805a1c1860b42b2d006df5d15"}
Oct 31 01:32:18.431535 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:18.431469    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2" event={"ID":"b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6","Type":"ContainerDied","Data":"94694effca29dbbad0fbda645b699535a02797e58e76bacd15f4827d252f9230"}
Oct 31 01:32:18.431535 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:18.431500    4476 scope.go:122] "RemoveContainer" containerID="c6869f7cdf7f88c0607292465de67e2dd34f17e805a1c1860b42b2d006df5d15"
Oct 31 01:32:18.441674 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:18.441653    4476 scope.go:122] "RemoveContainer" containerID="877ec2c7ca45181809ec11ff1fcab0c89938cb9e1a71c7a3a37e80fb57f85ea7"
Oct 31 01:32:18.450398 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:18.450277    4476 scope.go:122] "RemoveContainer" containerID="c6869f7cdf7f88c0607292465de67e2dd34f17e805a1c1860b42b2d006df5d15"
Oct 31 01:32:18.450587 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:32:18.450567    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6869f7cdf7f88c0607292465de67e2dd34f17e805a1c1860b42b2d006df5d15\": container with ID starting with c6869f7cdf7f88c0607292465de67e2dd34f17e805a1c1860b42b2d006df5d15 not found: ID does not exist" containerID="c6869f7cdf7f88c0607292465de67e2dd34f17e805a1c1860b42b2d006df5d15"
Oct 31 01:32:18.450658 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:18.450595    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6869f7cdf7f88c0607292465de67e2dd34f17e805a1c1860b42b2d006df5d15"} err="failed to get container status \"c6869f7cdf7f88c0607292465de67e2dd34f17e805a1c1860b42b2d006df5d15\": rpc error: code = NotFound desc = could not find container \"c6869f7cdf7f88c0607292465de67e2dd34f17e805a1c1860b42b2d006df5d15\": container with ID starting with c6869f7cdf7f88c0607292465de67e2dd34f17e805a1c1860b42b2d006df5d15 not found: ID does not exist"
Oct 31 01:32:18.450658 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:18.450614    4476 scope.go:122] "RemoveContainer" containerID="877ec2c7ca45181809ec11ff1fcab0c89938cb9e1a71c7a3a37e80fb57f85ea7"
Oct 31 01:32:18.450812 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:32:18.450792    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"877ec2c7ca45181809ec11ff1fcab0c89938cb9e1a71c7a3a37e80fb57f85ea7\": container with ID starting with 877ec2c7ca45181809ec11ff1fcab0c89938cb9e1a71c7a3a37e80fb57f85ea7 not found: ID does not exist" containerID="877ec2c7ca45181809ec11ff1fcab0c89938cb9e1a71c7a3a37e80fb57f85ea7"
Oct 31 01:32:18.450856 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:18.450819    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"877ec2c7ca45181809ec11ff1fcab0c89938cb9e1a71c7a3a37e80fb57f85ea7"} err="failed to get container status \"877ec2c7ca45181809ec11ff1fcab0c89938cb9e1a71c7a3a37e80fb57f85ea7\": rpc error: code = NotFound desc = could not find container \"877ec2c7ca45181809ec11ff1fcab0c89938cb9e1a71c7a3a37e80fb57f85ea7\": container with ID starting with 877ec2c7ca45181809ec11ff1fcab0c89938cb9e1a71c7a3a37e80fb57f85ea7 not found: ID does not exist"
Oct 31 01:32:18.594626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:18.594594    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"]
Oct 31 01:32:18.644319 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:18.644283    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7870b4-openshift-default-6cbc565675-8bcb2"]
Oct 31 01:32:20.328687 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:20.328650    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:20.454948 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:20.454917    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" path="/var/lib/kubelet/pods/b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6/volumes"
Oct 31 01:32:23.541603 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:23.541570    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:24.357041 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:24.356986    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:24.681209 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:24.681076    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:25.005591 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:25.005460    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:27.447017 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:27.446964    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:29.181505 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:32:29.181466    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:32:29.570387 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:29.570349    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:29.777233 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:29.777197    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:29.955903 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:29.955826    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:30.110207 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.110159    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"]
Oct 31 01:32:30.110441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.110417    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw" podUID="ff675b78-d5fa-4020-90d5-4ed8fa5162ce" containerName="discovery" containerID="cri-o://4b6be4cab381db9ca73ef79a0814687e76bd6a750bfcf81dca3b62f9eb3a0b60" gracePeriod=30
Oct 31 01:32:30.372800 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.372775    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:32:30.506480 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.506399    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-local-certs\") pod \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") "
Oct 31 01:32:30.506815 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.506791    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-csr-ca-configmap\") pod \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") "
Oct 31 01:32:30.507020 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.506999    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-kubeconfig\") pod \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") "
Oct 31 01:32:30.507168 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.507152    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-token\" (UniqueName: \"kubernetes.io/projected/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-token\") pod \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") "
Oct 31 01:32:30.507472 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.507451    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-csr-dns-cert\") pod \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") "
Oct 31 01:32:30.507634 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.507617    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-cacerts\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-cacerts\") pod \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") "
Oct 31 01:32:30.507826 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.507810    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-kube-api-access-nmxst\" (UniqueName: \"kubernetes.io/projected/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-kube-api-access-nmxst\") pod \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\" (UID: \"ff675b78-d5fa-4020-90d5-4ed8fa5162ce\") "
Oct 31 01:32:30.509500 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.509473    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-csr-ca-configmap" pod "ff675b78-d5fa-4020-90d5-4ed8fa5162ce" (UID: "ff675b78-d5fa-4020-90d5-4ed8fa5162ce"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:32:30.510236 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.509800    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-local-certs" pod "ff675b78-d5fa-4020-90d5-4ed8fa5162ce" (UID: "ff675b78-d5fa-4020-90d5-4ed8fa5162ce"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:30.511786 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.511749    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-csr-dns-cert" pod "ff675b78-d5fa-4020-90d5-4ed8fa5162ce" (UID: "ff675b78-d5fa-4020-90d5-4ed8fa5162ce"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:32:30.515245 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.515205    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-kubeconfig" pod "ff675b78-d5fa-4020-90d5-4ed8fa5162ce" (UID: "ff675b78-d5fa-4020-90d5-4ed8fa5162ce"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:32:30.515350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.515205    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-cacerts" pod "ff675b78-d5fa-4020-90d5-4ed8fa5162ce" (UID: "ff675b78-d5fa-4020-90d5-4ed8fa5162ce"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:32:30.515533 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.515511    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-kube-api-access-nmxst" pod "ff675b78-d5fa-4020-90d5-4ed8fa5162ce" (UID: "ff675b78-d5fa-4020-90d5-4ed8fa5162ce"). InnerVolumeSpecName "kube-api-access-nmxst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:30.517802 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.517651    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-token" pod "ff675b78-d5fa-4020-90d5-4ed8fa5162ce" (UID: "ff675b78-d5fa-4020-90d5-4ed8fa5162ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:30.518940 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.518915    4476 generic.go:356] "Generic (PLEG): container finished" podID="ff675b78-d5fa-4020-90d5-4ed8fa5162ce" containerID="4b6be4cab381db9ca73ef79a0814687e76bd6a750bfcf81dca3b62f9eb3a0b60" exitCode=0
Oct 31 01:32:30.519044 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.518998    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"
Oct 31 01:32:30.519044 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.519001    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw" event={"ID":"ff675b78-d5fa-4020-90d5-4ed8fa5162ce","Type":"ContainerDied","Data":"4b6be4cab381db9ca73ef79a0814687e76bd6a750bfcf81dca3b62f9eb3a0b60"}
Oct 31 01:32:30.519162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.519081    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw" event={"ID":"ff675b78-d5fa-4020-90d5-4ed8fa5162ce","Type":"ContainerDied","Data":"989449e7d38606f3e3dd18876b2738c5497ebe01b33d0a924840726cdd496330"}
Oct 31 01:32:30.519162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.519110    4476 scope.go:122] "RemoveContainer" containerID="4b6be4cab381db9ca73ef79a0814687e76bd6a750bfcf81dca3b62f9eb3a0b60"
Oct 31 01:32:30.548237 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.547647    4476 scope.go:122] "RemoveContainer" containerID="4b6be4cab381db9ca73ef79a0814687e76bd6a750bfcf81dca3b62f9eb3a0b60"
Oct 31 01:32:30.548237 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:32:30.548162    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b6be4cab381db9ca73ef79a0814687e76bd6a750bfcf81dca3b62f9eb3a0b60\": container with ID starting with 4b6be4cab381db9ca73ef79a0814687e76bd6a750bfcf81dca3b62f9eb3a0b60 not found: ID does not exist" containerID="4b6be4cab381db9ca73ef79a0814687e76bd6a750bfcf81dca3b62f9eb3a0b60"
Oct 31 01:32:30.548381 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.548236    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b6be4cab381db9ca73ef79a0814687e76bd6a750bfcf81dca3b62f9eb3a0b60"} err="failed to get container status \"4b6be4cab381db9ca73ef79a0814687e76bd6a750bfcf81dca3b62f9eb3a0b60\": rpc error: code = NotFound desc = could not find container \"4b6be4cab381db9ca73ef79a0814687e76bd6a750bfcf81dca3b62f9eb3a0b60\": container with ID starting with 4b6be4cab381db9ca73ef79a0814687e76bd6a750bfcf81dca3b62f9eb3a0b60 not found: ID does not exist"
Oct 31 01:32:30.610089 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.610060    4476 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-csr-dns-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:30.610089 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.610088    4476 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-cacerts\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:30.610285 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.610099    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmxst\" (UniqueName: \"kubernetes.io/projected/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-kube-api-access-nmxst\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:30.610285 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.610109    4476 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-local-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:30.610285 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.610118    4476 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-csr-ca-configmap\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:30.610285 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.610126    4476 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-kubeconfig\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:30.610285 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.610135    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff675b78-d5fa-4020-90d5-4ed8fa5162ce-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:30.682812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.682779    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"]
Oct 31 01:32:30.687304 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.687273    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhmdw"]
Oct 31 01:32:30.903311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:30.903288    4476 scope.go:122] "RemoveContainer" containerID="6c5e2c85883bae029cef811cf937dac3ef6eb2cc8bb451097fb2fc75f8ade432"
Oct 31 01:32:32.454239 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:32.454203    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff675b78-d5fa-4020-90d5-4ed8fa5162ce" path="/var/lib/kubelet/pods/ff675b78-d5fa-4020-90d5-4ed8fa5162ce/volumes"
Oct 31 01:32:33.466040 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:33.466009    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"]
Oct 31 01:32:33.466523 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:33.466276    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9" podUID="eb725fb1-db73-4ca4-9dba-d4c2028f79e6" containerName="istio-proxy" containerID="cri-o://2b943ade8183ca57c83f964924daef6fc10edf7eefe4923b0ea414af0069052e" gracePeriod=30
Oct 31 01:32:34.445900 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:34.445865    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:36.982240 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:36.982208    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:38.723461 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.723434    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:32:38.900824 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.900794    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-token\" (UniqueName: \"kubernetes.io/projected/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-token\") pod \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") "
Oct 31 01:32:38.900824 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.900827    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-workload-certs\") pod \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") "
Oct 31 01:32:38.901090 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.900893    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-credential-socket\") pod \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") "
Oct 31 01:32:38.901090 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.900986    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-data\") pod \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") "
Oct 31 01:32:38.901090 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.901042    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-podinfo\") pod \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") "
Oct 31 01:32:38.901309 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.901097    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-envoy\") pod \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") "
Oct 31 01:32:38.901309 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.901191    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-workload-certs" pod "eb725fb1-db73-4ca4-9dba-d4c2028f79e6" (UID: "eb725fb1-db73-4ca4-9dba-d4c2028f79e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:38.901309 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.901205    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-credential-socket" pod "eb725fb1-db73-4ca4-9dba-d4c2028f79e6" (UID: "eb725fb1-db73-4ca4-9dba-d4c2028f79e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:38.901309 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.901261    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-kube-api-access-rffp2\" (UniqueName: \"kubernetes.io/projected/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-kube-api-access-rffp2\") pod \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") "
Oct 31 01:32:38.901309 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.901297    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-workload-socket\") pod \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") "
Oct 31 01:32:38.901581 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.901328    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istiod-ca-cert\") pod \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\" (UID: \"eb725fb1-db73-4ca4-9dba-d4c2028f79e6\") "
Oct 31 01:32:38.901581 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.901380    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-data" pod "eb725fb1-db73-4ca4-9dba-d4c2028f79e6" (UID: "eb725fb1-db73-4ca4-9dba-d4c2028f79e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:38.901691 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.901666    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istiod-ca-cert" pod "eb725fb1-db73-4ca4-9dba-d4c2028f79e6" (UID: "eb725fb1-db73-4ca4-9dba-d4c2028f79e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:32:38.902125 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.902015    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:38.902125 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.902043    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:38.902125 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.902049    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-workload-socket" pod "eb725fb1-db73-4ca4-9dba-d4c2028f79e6" (UID: "eb725fb1-db73-4ca4-9dba-d4c2028f79e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:38.902125 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.902058    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:38.902125 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.902095    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:38.903954 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.903928    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-envoy" pod "eb725fb1-db73-4ca4-9dba-d4c2028f79e6" (UID: "eb725fb1-db73-4ca4-9dba-d4c2028f79e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:38.904055 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.903950    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-podinfo" pod "eb725fb1-db73-4ca4-9dba-d4c2028f79e6" (UID: "eb725fb1-db73-4ca4-9dba-d4c2028f79e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:32:38.904055 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.903955    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-token" pod "eb725fb1-db73-4ca4-9dba-d4c2028f79e6" (UID: "eb725fb1-db73-4ca4-9dba-d4c2028f79e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:38.904289 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.904268    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-kube-api-access-rffp2" pod "eb725fb1-db73-4ca4-9dba-d4c2028f79e6" (UID: "eb725fb1-db73-4ca4-9dba-d4c2028f79e6"). InnerVolumeSpecName "kube-api-access-rffp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:38.925815 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:38.925795    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:39.003554 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.003526    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:39.003554 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.003550    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:39.003707 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.003560    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:39.003707 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.003570    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rffp2\" (UniqueName: \"kubernetes.io/projected/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-kube-api-access-rffp2\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:39.003707 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.003578    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb725fb1-db73-4ca4-9dba-d4c2028f79e6-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:39.135407 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.135367    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:39.322321 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.322288    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:39.517010 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.516979    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:39.567403 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.567366    4476 generic.go:356] "Generic (PLEG): container finished" podID="eb725fb1-db73-4ca4-9dba-d4c2028f79e6" containerID="2b943ade8183ca57c83f964924daef6fc10edf7eefe4923b0ea414af0069052e" exitCode=0
Oct 31 01:32:39.567563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.567419    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"
Oct 31 01:32:39.567563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.567446    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9" event={"ID":"eb725fb1-db73-4ca4-9dba-d4c2028f79e6","Type":"ContainerDied","Data":"2b943ade8183ca57c83f964924daef6fc10edf7eefe4923b0ea414af0069052e"}
Oct 31 01:32:39.567563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.567503    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9" event={"ID":"eb725fb1-db73-4ca4-9dba-d4c2028f79e6","Type":"ContainerDied","Data":"0c35a0c873cbb19318bd257ae1cce910c8e0b89a9f8dec2fb7f6623f3f7de5de"}
Oct 31 01:32:39.567563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.567524    4476 scope.go:122] "RemoveContainer" containerID="2b943ade8183ca57c83f964924daef6fc10edf7eefe4923b0ea414af0069052e"
Oct 31 01:32:39.577953 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.577935    4476 scope.go:122] "RemoveContainer" containerID="2b943ade8183ca57c83f964924daef6fc10edf7eefe4923b0ea414af0069052e"
Oct 31 01:32:39.578206 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:32:39.578170    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b943ade8183ca57c83f964924daef6fc10edf7eefe4923b0ea414af0069052e\": container with ID starting with 2b943ade8183ca57c83f964924daef6fc10edf7eefe4923b0ea414af0069052e not found: ID does not exist" containerID="2b943ade8183ca57c83f964924daef6fc10edf7eefe4923b0ea414af0069052e"
Oct 31 01:32:39.578274 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.578220    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b943ade8183ca57c83f964924daef6fc10edf7eefe4923b0ea414af0069052e"} err="failed to get container status \"2b943ade8183ca57c83f964924daef6fc10edf7eefe4923b0ea414af0069052e\": rpc error: code = NotFound desc = could not find container \"2b943ade8183ca57c83f964924daef6fc10edf7eefe4923b0ea414af0069052e\": container with ID starting with 2b943ade8183ca57c83f964924daef6fc10edf7eefe4923b0ea414af0069052e not found: ID does not exist"
Oct 31 01:32:39.782246 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.782212    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"]
Oct 31 01:32:39.832112 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:39.832047    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-9bgk9"]
Oct 31 01:32:40.453961 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:40.453928    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb725fb1-db73-4ca4-9dba-d4c2028f79e6" path="/var/lib/kubelet/pods/eb725fb1-db73-4ca4-9dba-d4c2028f79e6/volumes"
Oct 31 01:32:42.123309 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.123252    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:42.352072 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.352035    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:42.484293 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484149    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"]
Oct 31 01:32:42.484744 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484724    4476 cpu_manager.go:414] "containerMap: removing container" podUID="7f997010-2ff4-4bed-8cac-efae308885a3" containerName="payload-pre-processing"
Oct 31 01:32:42.484847 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484746    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f997010-2ff4-4bed-8cac-efae308885a3" containerName="payload-pre-processing"
Oct 31 01:32:42.484847 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484767    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c979dc7a-14aa-4a28-a1c9-415265b7e48b" containerName="cleanup"
Oct 31 01:32:42.484847 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484774    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c979dc7a-14aa-4a28-a1c9-415265b7e48b" containerName="cleanup"
Oct 31 01:32:42.484847 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484806    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5ad11508-ee1b-404c-9b04-93db72bfb27a" containerName="payload-processing"
Oct 31 01:32:42.484847 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484813    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad11508-ee1b-404c-9b04-93db72bfb27a" containerName="payload-processing"
Oct 31 01:32:42.484847 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484826    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerName="istio-proxy"
Oct 31 01:32:42.484847 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484833    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerName="istio-proxy"
Oct 31 01:32:42.484847 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484843    4476 cpu_manager.go:414] "containerMap: removing container" podUID="eb725fb1-db73-4ca4-9dba-d4c2028f79e6" containerName="istio-proxy"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484850    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb725fb1-db73-4ca4-9dba-d4c2028f79e6" containerName="istio-proxy"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484866    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe" containerName="payload-pre-processing"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484874    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe" containerName="payload-pre-processing"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484884    4476 cpu_manager.go:414] "containerMap: removing container" podUID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484891    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484905    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c979dc7a-14aa-4a28-a1c9-415265b7e48b" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484912    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c979dc7a-14aa-4a28-a1c9-415265b7e48b" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484926    4476 cpu_manager.go:414] "containerMap: removing container" podUID="7840d177-98d4-4dc7-9b08-18dd2e6aa545" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484933    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7840d177-98d4-4dc7-9b08-18dd2e6aa545" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484945    4476 cpu_manager.go:414] "containerMap: removing container" podUID="7dc706e1-fe7c-4e88-88a2-ef35706e2791" containerName="istio-proxy"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484953    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc706e1-fe7c-4e88-88a2-ef35706e2791" containerName="istio-proxy"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484963    4476 cpu_manager.go:414] "containerMap: removing container" podUID="7840d177-98d4-4dc7-9b08-18dd2e6aa545" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484970    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7840d177-98d4-4dc7-9b08-18dd2e6aa545" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484979    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ff675b78-d5fa-4020-90d5-4ed8fa5162ce" containerName="discovery"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484986    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff675b78-d5fa-4020-90d5-4ed8fa5162ce" containerName="discovery"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.484996    4476 cpu_manager.go:414] "containerMap: removing container" podUID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485003    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485017    4476 cpu_manager.go:414] "containerMap: removing container" podUID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485024    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485044    4476 cpu_manager.go:414] "containerMap: removing container" podUID="6ae7c551-8aa2-4728-84ab-c12f662e8387" containerName="payload-processing"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485051    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae7c551-8aa2-4728-84ab-c12f662e8387" containerName="payload-processing"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485264    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc706e1-fe7c-4e88-88a2-ef35706e2791" containerName="istio-proxy"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485284    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7840d177-98d4-4dc7-9b08-18dd2e6aa545" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485294    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff675b78-d5fa-4020-90d5-4ed8fa5162ce" containerName="discovery"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485304    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485313    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485325    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7840d177-98d4-4dc7-9b08-18dd2e6aa545" containerName="cleanup"
Oct 31 01:32:42.485326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485334    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae7c551-8aa2-4728-84ab-c12f662e8387" containerName="payload-processing"
Oct 31 01:32:42.486975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485344    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerName="istio-proxy"
Oct 31 01:32:42.486975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485353    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f997010-2ff4-4bed-8cac-efae308885a3" containerName="payload-pre-processing"
Oct 31 01:32:42.486975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485361    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c979dc7a-14aa-4a28-a1c9-415265b7e48b" containerName="cleanup"
Oct 31 01:32:42.486975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485375    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad11508-ee1b-404c-9b04-93db72bfb27a" containerName="payload-processing"
Oct 31 01:32:42.486975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485384    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb725fb1-db73-4ca4-9dba-d4c2028f79e6" containerName="istio-proxy"
Oct 31 01:32:42.486975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.485393    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56d5228-1d0f-4fd2-b0e0-f6f475ae2cbe" containerName="payload-pre-processing"
Oct 31 01:32:42.490507 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.490487    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"
Oct 31 01:32:42.542889 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.542863    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcf79\" (UniqueName: \"kubernetes.io/projected/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f-kube-api-access-mcf79\") pod \"payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf\" (UID: \"ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"
Oct 31 01:32:42.543260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.543234    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf\" (UID: \"ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"
Oct 31 01:32:42.546542 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.546521    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"]
Oct 31 01:32:42.592925 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.592902    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c9da8f\""
Oct 31 01:32:42.646935 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.646843    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf\" (UID: \"ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"
Oct 31 01:32:42.646935 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.646908    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcf79\" (UniqueName: \"kubernetes.io/projected/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f-kube-api-access-mcf79\") pod \"payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf\" (UID: \"ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"
Oct 31 01:32:42.647822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.647794    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf\" (UID: \"ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"
Oct 31 01:32:42.704354 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.704320    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcf79\" (UniqueName: \"kubernetes.io/projected/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f-kube-api-access-mcf79\") pod \"payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf\" (UID: \"ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"
Oct 31 01:32:42.803432 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:42.803395    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"
Oct 31 01:32:43.178224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:43.178138    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"]
Oct 31 01:32:43.182496 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:32:43.182463    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4170e4_5efa_456d_b6f4_a16d2f2d1f9f.slice/crio-6e1e6a63e7078bc74101ea84dedc3348c8e4203e4b3a84fcbef56d1c0db3681a WatchSource:0}: Error finding container 6e1e6a63e7078bc74101ea84dedc3348c8e4203e4b3a84fcbef56d1c0db3681a: Status 404 returned error can't find the container with id 6e1e6a63e7078bc74101ea84dedc3348c8e4203e4b3a84fcbef56d1c0db3681a
Oct 31 01:32:43.591759 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:43.591720    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf" event={"ID":"ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f","Type":"ContainerStarted","Data":"e0c3a26f4aad15bbe4772cb3aa801c3079dbe1789e78a048552a31c807173f61"}
Oct 31 01:32:43.591965 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:43.591773    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf" event={"ID":"ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f","Type":"ContainerStarted","Data":"6e1e6a63e7078bc74101ea84dedc3348c8e4203e4b3a84fcbef56d1c0db3681a"}
Oct 31 01:32:43.591965 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:43.591916    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"
Oct 31 01:32:43.706060 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:43.706010    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf" podStartSLOduration=1.705997669 podStartE2EDuration="1.705997669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:32:43.703302568 +0000 UTC m=+2173.393185696" watchObservedRunningTime="2026-10-31 01:32:43.705997669 +0000 UTC m=+2173.395880812"
Oct 31 01:32:44.695123 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:44.695067    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:47.670505 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:47.670472    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:49.060364 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:49.060328    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:49.260426 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:49.260386    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:49.463314 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:49.463232    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:49.842660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:49.842603    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:53.410984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.410948    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:53.905708 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.905672    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"]
Oct 31 01:32:53.906203 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.906189    4476 cpu_manager.go:414] "containerMap: removing container" podUID="7840d177-98d4-4dc7-9b08-18dd2e6aa545" containerName="cleanup"
Oct 31 01:32:53.906257 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.906204    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7840d177-98d4-4dc7-9b08-18dd2e6aa545" containerName="cleanup"
Oct 31 01:32:53.906257 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.906240    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerName="istio-proxy"
Oct 31 01:32:53.906257 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.906244    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerName="istio-proxy"
Oct 31 01:32:53.906355 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.906314    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f675bd-de2e-4bc7-b01a-adcacbb50d8f" containerName="cleanup"
Oct 31 01:32:53.906355 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.906324    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7840d177-98d4-4dc7-9b08-18dd2e6aa545" containerName="cleanup"
Oct 31 01:32:53.906355 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.906329    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88a32b2-e0a2-4d5d-ad6c-e079ba60c5b6" containerName="istio-proxy"
Oct 31 01:32:53.906355 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.906336    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c979dc7a-14aa-4a28-a1c9-415265b7e48b" containerName="cleanup"
Oct 31 01:32:53.909694 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.909679    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:53.965707 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.965668    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-workload-socket\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:53.987729 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.965932    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b3025-1c9e-4c04-9beb-a21c93225c0f-istiod-ca-cert\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:53.988149 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.988123    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-podinfo\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:53.988354 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.988317    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-data\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:53.988354 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.988351    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46njq\" (UniqueName: \"kubernetes.io/projected/185b3025-1c9e-4c04-9beb-a21c93225c0f-kube-api-access-46njq\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:53.988539 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.988409    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-token\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:53.988539 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.988446    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-envoy\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:53.988539 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.988491    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-credential-socket\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:53.988539 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.988519    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-workload-certs\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:53.988862 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:53.969773    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6c7db9-openshift-default-dockercfg-v4667\""
Oct 31 01:32:54.017550 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.017518    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"]
Oct 31 01:32:54.083411 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.083376    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:54.089406 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.089382    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-token\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.089568 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.089417    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-envoy\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.089568 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.089451    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-credential-socket\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.089568 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.089468    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-workload-certs\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.089568 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.089543    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-workload-socket\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.089568 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.089560    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b3025-1c9e-4c04-9beb-a21c93225c0f-istiod-ca-cert\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.089836 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.089649    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-podinfo\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.089836 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.089668    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-data\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.089836 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.089691    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46njq\" (UniqueName: \"kubernetes.io/projected/185b3025-1c9e-4c04-9beb-a21c93225c0f-kube-api-access-46njq\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.090531 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.090209    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-workload-socket\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.090660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.090615    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b3025-1c9e-4c04-9beb-a21c93225c0f-istiod-ca-cert\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.091050 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.090856    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-data\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.091050 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.091010    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-workload-certs\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.091529 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.091500    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-credential-socket\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.092991 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.092968    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-podinfo\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.093428 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.093406    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-envoy\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.147005 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.146979    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-token\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.147165 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.147146    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46njq\" (UniqueName: \"kubernetes.io/projected/185b3025-1c9e-4c04-9beb-a21c93225c0f-kube-api-access-46njq\") pod \"e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") " pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.222364 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.222307    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:54.433112 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.433073    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:54.599809 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.599782    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"
Oct 31 01:32:54.649233 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.649207    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"]
Oct 31 01:32:54.650475 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:32:54.650448    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185b3025_1c9e_4c04_9beb_a21c93225c0f.slice/crio-6d9603048c06be9b14299363afc4465669b35b6da4436cb750d208e5cc0b758a WatchSource:0}: Error finding container 6d9603048c06be9b14299363afc4465669b35b6da4436cb750d208e5cc0b758a: Status 404 returned error can't find the container with id 6d9603048c06be9b14299363afc4465669b35b6da4436cb750d208e5cc0b758a
Oct 31 01:32:54.652903 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.652872    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:32:54.652987 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.652936    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:32:54.652987 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.652965    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:32:54.673318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:54.673151    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:55.662618 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:55.662583    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv" event={"ID":"185b3025-1c9e-4c04-9beb-a21c93225c0f","Type":"ContainerStarted","Data":"982d584be2c8f70db2d53a525d56dae4c1a02640f0084d0445ecdee832916e4e"}
Oct 31 01:32:55.663056 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:55.662638    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv" event={"ID":"185b3025-1c9e-4c04-9beb-a21c93225c0f","Type":"ContainerStarted","Data":"6d9603048c06be9b14299363afc4465669b35b6da4436cb750d208e5cc0b758a"}
Oct 31 01:32:55.831841 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:55.831787    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv" podStartSLOduration=2.8317740860000002 podStartE2EDuration="2.831774086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:32:55.828046771 +0000 UTC m=+2185.517929899" watchObservedRunningTime="2026-10-31 01:32:55.831774086 +0000 UTC m=+2185.521657212"
Oct 31 01:32:56.223295 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:56.223267    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:56.228534 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:56.228510    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:56.667659 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:56.667634    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:56.668562 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:56.668544    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:32:58.676753 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:58.676713    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:58.878067 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:58.878037    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:59.077857 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:59.077816    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:59.182208 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:32:59.182150    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:32:59.793190 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:32:59.793149    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:02.633646 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:02.633597    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:02.820837 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:02.820800    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:04.346396 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:04.346352    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:04.924028 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:04.923997    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:05.266216 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:05.266113    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:05.898510 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:05.898471    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:06.118360 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:06.118326    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:08.652002 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:08.651968    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:08.824793 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:08.824764    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:09.879946 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:09.879916    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:14.611434 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:14.611351    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:18.485854 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:18.485814    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"]
Oct 31 01:33:18.486326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:18.485996    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m" podUID="e65056fe-146a-4d91-ac90-964f71f943a7" containerName="istio-proxy" containerID="cri-o://3fee524746cf85c00dd014cf730d607010e25f2f848dcfb099b8e7dcc889ed17" gracePeriod=30
Oct 31 01:33:18.499731 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:18.499674    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:18.788484 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:18.788453    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:19.590070 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:19.590026    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:20.555809 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:20.555773    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:21.484987 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:21.484954    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:22.661863 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:22.661818    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:22.855836 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:22.855804    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:23.625127 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.625097    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:33:23.625289 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.625155    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m" podUID="e65056fe-146a-4d91-ac90-964f71f943a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 01:33:23.756560 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.756503    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:33:23.836701 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.836673    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-credential-socket\") pod \"e65056fe-146a-4d91-ac90-964f71f943a7\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") "
Oct 31 01:33:23.836882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.836718    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-istio-envoy\") pod \"e65056fe-146a-4d91-ac90-964f71f943a7\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") "
Oct 31 01:33:23.836882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.836772    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e65056fe-146a-4d91-ac90-964f71f943a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65056fe-146a-4d91-ac90-964f71f943a7-istio-podinfo\") pod \"e65056fe-146a-4d91-ac90-964f71f943a7\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") "
Oct 31 01:33:23.836882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.836823    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-workload-certs\") pod \"e65056fe-146a-4d91-ac90-964f71f943a7\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") "
Oct 31 01:33:23.836882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.836869    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65056fe-146a-4d91-ac90-964f71f943a7-kube-api-access-tmrlk\" (UniqueName: \"kubernetes.io/projected/e65056fe-146a-4d91-ac90-964f71f943a7-kube-api-access-tmrlk\") pod \"e65056fe-146a-4d91-ac90-964f71f943a7\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") "
Oct 31 01:33:23.837103 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.836899    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65056fe-146a-4d91-ac90-964f71f943a7-istio-token\" (UniqueName: \"kubernetes.io/projected/e65056fe-146a-4d91-ac90-964f71f943a7-istio-token\") pod \"e65056fe-146a-4d91-ac90-964f71f943a7\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") "
Oct 31 01:33:23.837103 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.836941    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-workload-socket\") pod \"e65056fe-146a-4d91-ac90-964f71f943a7\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") "
Oct 31 01:33:23.837103 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.836967    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-credential-socket" pod "e65056fe-146a-4d91-ac90-964f71f943a7" (UID: "e65056fe-146a-4d91-ac90-964f71f943a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:23.837103 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.836980    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e65056fe-146a-4d91-ac90-964f71f943a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65056fe-146a-4d91-ac90-964f71f943a7-istiod-ca-cert\") pod \"e65056fe-146a-4d91-ac90-964f71f943a7\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") "
Oct 31 01:33:23.837361 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.837213    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-istio-data\") pod \"e65056fe-146a-4d91-ac90-964f71f943a7\" (UID: \"e65056fe-146a-4d91-ac90-964f71f943a7\") "
Oct 31 01:33:23.837361 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.837308    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65056fe-146a-4d91-ac90-964f71f943a7-istiod-ca-cert" pod "e65056fe-146a-4d91-ac90-964f71f943a7" (UID: "e65056fe-146a-4d91-ac90-964f71f943a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:23.837459 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.837389    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-workload-certs" pod "e65056fe-146a-4d91-ac90-964f71f943a7" (UID: "e65056fe-146a-4d91-ac90-964f71f943a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:23.837682 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.837584    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-workload-socket" pod "e65056fe-146a-4d91-ac90-964f71f943a7" (UID: "e65056fe-146a-4d91-ac90-964f71f943a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:23.837682 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.837604    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-istio-data" pod "e65056fe-146a-4d91-ac90-964f71f943a7" (UID: "e65056fe-146a-4d91-ac90-964f71f943a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:23.837862 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.837831    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:23.837862 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.837853    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:23.838089 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.838049    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:23.838232 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.838091    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:23.838232 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.838109    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65056fe-146a-4d91-ac90-964f71f943a7-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:23.839747 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.839702    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65056fe-146a-4d91-ac90-964f71f943a7-istio-token" pod "e65056fe-146a-4d91-ac90-964f71f943a7" (UID: "e65056fe-146a-4d91-ac90-964f71f943a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:23.839853 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.839803    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-istio-envoy" pod "e65056fe-146a-4d91-ac90-964f71f943a7" (UID: "e65056fe-146a-4d91-ac90-964f71f943a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:23.839853 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.839805    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e65056fe-146a-4d91-ac90-964f71f943a7-istio-podinfo" pod "e65056fe-146a-4d91-ac90-964f71f943a7" (UID: "e65056fe-146a-4d91-ac90-964f71f943a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:33:23.840216 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.840187    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65056fe-146a-4d91-ac90-964f71f943a7-kube-api-access-tmrlk" pod "e65056fe-146a-4d91-ac90-964f71f943a7" (UID: "e65056fe-146a-4d91-ac90-964f71f943a7"). InnerVolumeSpecName "kube-api-access-tmrlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:23.854003 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.853975    4476 generic.go:356] "Generic (PLEG): container finished" podID="e65056fe-146a-4d91-ac90-964f71f943a7" containerID="3fee524746cf85c00dd014cf730d607010e25f2f848dcfb099b8e7dcc889ed17" exitCode=0
Oct 31 01:33:23.854112 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.854024    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"
Oct 31 01:33:23.854112 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.854044    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m" event={"ID":"e65056fe-146a-4d91-ac90-964f71f943a7","Type":"ContainerDied","Data":"3fee524746cf85c00dd014cf730d607010e25f2f848dcfb099b8e7dcc889ed17"}
Oct 31 01:33:23.854112 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.854097    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m" event={"ID":"e65056fe-146a-4d91-ac90-964f71f943a7","Type":"ContainerDied","Data":"1e242313339913103bc2f6fd90eef409f2f6eb10eced0f6472b674b17d307f07"}
Oct 31 01:33:23.854329 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.854126    4476 scope.go:122] "RemoveContainer" containerID="3fee524746cf85c00dd014cf730d607010e25f2f848dcfb099b8e7dcc889ed17"
Oct 31 01:33:23.863660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.863643    4476 scope.go:122] "RemoveContainer" containerID="3fee524746cf85c00dd014cf730d607010e25f2f848dcfb099b8e7dcc889ed17"
Oct 31 01:33:23.863919 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:33:23.863904    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fee524746cf85c00dd014cf730d607010e25f2f848dcfb099b8e7dcc889ed17\": container with ID starting with 3fee524746cf85c00dd014cf730d607010e25f2f848dcfb099b8e7dcc889ed17 not found: ID does not exist" containerID="3fee524746cf85c00dd014cf730d607010e25f2f848dcfb099b8e7dcc889ed17"
Oct 31 01:33:23.863968 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.863926    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fee524746cf85c00dd014cf730d607010e25f2f848dcfb099b8e7dcc889ed17"} err="failed to get container status \"3fee524746cf85c00dd014cf730d607010e25f2f848dcfb099b8e7dcc889ed17\": rpc error: code = NotFound desc = could not find container \"3fee524746cf85c00dd014cf730d607010e25f2f848dcfb099b8e7dcc889ed17\": container with ID starting with 3fee524746cf85c00dd014cf730d607010e25f2f848dcfb099b8e7dcc889ed17 not found: ID does not exist"
Oct 31 01:33:23.939313 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.939281    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65056fe-146a-4d91-ac90-964f71f943a7-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:23.939313 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.939304    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65056fe-146a-4d91-ac90-964f71f943a7-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:23.939313 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.939316    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmrlk\" (UniqueName: \"kubernetes.io/projected/e65056fe-146a-4d91-ac90-964f71f943a7-kube-api-access-tmrlk\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:23.939603 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:23.939325    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65056fe-146a-4d91-ac90-964f71f943a7-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:24.085635 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.085599    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"]
Oct 31 01:33:24.090698 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.090672    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-pqz7m"]
Oct 31 01:33:24.116511 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.116474    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:24.312136 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.312104    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:24.379911 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.379843    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"]
Oct 31 01:33:24.380420 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.380403    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e65056fe-146a-4d91-ac90-964f71f943a7" containerName="istio-proxy"
Oct 31 01:33:24.380513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.380422    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65056fe-146a-4d91-ac90-964f71f943a7" containerName="istio-proxy"
Oct 31 01:33:24.380551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.380520    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65056fe-146a-4d91-ac90-964f71f943a7" containerName="istio-proxy"
Oct 31 01:33:24.383626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.383612    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:24.435894 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.435870    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c9da8f-serving-cert\""
Oct 31 01:33:24.443076 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.443043    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"]
Oct 31 01:33:24.445829 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.445802    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:24.445981 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.445962    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdq54\" (UniqueName: \"kubernetes.io/projected/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-kube-api-access-tdq54\") pod \"maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:24.446047 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.446033    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:24.446114 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.446092    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-metrics-certs\") pod \"maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:24.456896 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.456873    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e65056fe-146a-4d91-ac90-964f71f943a7" path="/var/lib/kubelet/pods/e65056fe-146a-4d91-ac90-964f71f943a7/volumes"
Oct 31 01:33:24.547522 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.547496    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:24.547672 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:33:24.547628    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-serving-cert: secret "maas-api-e2e-ipp-a-c9da8f-serving-cert" not found
Oct 31 01:33:24.547672 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.547632    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdq54\" (UniqueName: \"kubernetes.io/projected/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-kube-api-access-tdq54\") pod \"maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:24.547805 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:33:24.547689    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-maas-api-tls podName:c8a86b81-5e45-4c08-9d7c-9f1b8f11a201 nodeName:}" failed. No retries permitted until 2026-10-31 01:33:25.047670529 +0000 UTC m=+2214.737553655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-maas-api-tls") pod "maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9" (UID: "c8a86b81-5e45-4c08-9d7c-9f1b8f11a201") : secret "maas-api-e2e-ipp-a-c9da8f-serving-cert" not found
Oct 31 01:33:24.547805 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.547726    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:24.547805 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.547775    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-metrics-certs\") pod \"maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:24.548536 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.548512    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:24.550456 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.550435    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-metrics-certs\") pod \"maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:24.606790 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.606758    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdq54\" (UniqueName: \"kubernetes.io/projected/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-kube-api-access-tdq54\") pod \"maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:24.616496 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.616474    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:24.803478 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.803450    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:24.983324 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:24.983289    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:25.055412 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:25.055336    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:25.057867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:25.057840    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:25.296771 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:25.296738    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:25.686275 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:25.686252    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"]
Oct 31 01:33:25.687925 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:33:25.687898    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a86b81_5e45_4c08_9d7c_9f1b8f11a201.slice/crio-38868c994398e6c1e7f6d3467ce723e789e803a33d1e4e1a219b957e3bd582ff WatchSource:0}: Error finding container 38868c994398e6c1e7f6d3467ce723e789e803a33d1e4e1a219b957e3bd582ff: Status 404 returned error can't find the container with id 38868c994398e6c1e7f6d3467ce723e789e803a33d1e4e1a219b957e3bd582ff
Oct 31 01:33:25.872188 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:25.872139    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9" event={"ID":"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201","Type":"ContainerStarted","Data":"38868c994398e6c1e7f6d3467ce723e789e803a33d1e4e1a219b957e3bd582ff"}
Oct 31 01:33:26.879549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:26.879456    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9" event={"ID":"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201","Type":"ContainerStarted","Data":"a9169e3dacedcad9a34ece18477f80371b25604a1e192a888ba8676175e7e133"}
Oct 31 01:33:26.879966 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:26.879653    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:26.996145 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:26.996090    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9" podStartSLOduration=2.070240708 podStartE2EDuration="2.996071812s" totalImagesPullingTime="925.831104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:26.993260264 +0000 UTC m=+2216.683143392" watchObservedRunningTime="2026-10-31 01:33:26.996071812 +0000 UTC m=+2216.685954939"
Oct 31 01:33:27.404095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:27.404061    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:27.824731 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:27.824688    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:29.181614 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:33:29.181577    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:33:29.718382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:29.718328    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:31.840897 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:31.840853    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:32.085888 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:32.085823    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:32.282664 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:32.282637    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:32.495352 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:32.495320    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:32.891838 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:32.891806    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:33:33.257144 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:33.257067    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:33.754307 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:33.754270    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:36.299786 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:36.299752    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:37.318497 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:37.318464    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:41.065005 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:41.064968    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:46.096296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:46.096255    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:46.577981 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:46.577951    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:46.784440 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:46.784403    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:48.024974 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:48.024934    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:49.742835 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:49.742798    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:50.038465 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.038427    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:50.148726 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.148692    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"]
Oct 31 01:33:50.153801 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.153783    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"
Oct 31 01:33:50.209148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.209111    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"]
Oct 31 01:33:50.250204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.250147    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-726vg\" (UniqueName: \"kubernetes.io/projected/396b763c-807d-4f81-9bbc-20ba4696fee4-kube-api-access-726vg\") pod \"payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2\" (UID: \"396b763c-807d-4f81-9bbc-20ba4696fee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"
Oct 31 01:33:50.250661 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.250625    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396b763c-807d-4f81-9bbc-20ba4696fee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2\" (UID: \"396b763c-807d-4f81-9bbc-20ba4696fee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"
Oct 31 01:33:50.254473 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.254450    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c0359f-dockercfg-ffs56\""
Oct 31 01:33:50.254671 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.254639    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c0359f\""
Oct 31 01:33:50.266688 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.266657    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"]
Oct 31 01:33:50.270627 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.270603    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"
Oct 31 01:33:50.326153 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.326081    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"]
Oct 31 01:33:50.351610 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.351577    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396b763c-807d-4f81-9bbc-20ba4696fee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2\" (UID: \"396b763c-807d-4f81-9bbc-20ba4696fee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"
Oct 31 01:33:50.351787 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.351665    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-726vg\" (UniqueName: \"kubernetes.io/projected/396b763c-807d-4f81-9bbc-20ba4696fee4-kube-api-access-726vg\") pod \"payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2\" (UID: \"396b763c-807d-4f81-9bbc-20ba4696fee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"
Oct 31 01:33:50.351787 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.351695    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp7zp\" (UniqueName: \"kubernetes.io/projected/ddcfb701-535b-4321-94e2-d64efff7d9ef-kube-api-access-pp7zp\") pod \"payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58\" (UID: \"ddcfb701-535b-4321-94e2-d64efff7d9ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"
Oct 31 01:33:50.351787 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.351717    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcfb701-535b-4321-94e2-d64efff7d9ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58\" (UID: \"ddcfb701-535b-4321-94e2-d64efff7d9ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"
Oct 31 01:33:50.352393 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.352370    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396b763c-807d-4f81-9bbc-20ba4696fee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2\" (UID: \"396b763c-807d-4f81-9bbc-20ba4696fee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"
Oct 31 01:33:50.408571 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.408543    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-726vg\" (UniqueName: \"kubernetes.io/projected/396b763c-807d-4f81-9bbc-20ba4696fee4-kube-api-access-726vg\") pod \"payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2\" (UID: \"396b763c-807d-4f81-9bbc-20ba4696fee4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"
Oct 31 01:33:50.452655 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.452606    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp7zp\" (UniqueName: \"kubernetes.io/projected/ddcfb701-535b-4321-94e2-d64efff7d9ef-kube-api-access-pp7zp\") pod \"payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58\" (UID: \"ddcfb701-535b-4321-94e2-d64efff7d9ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"
Oct 31 01:33:50.452655 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.452651    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcfb701-535b-4321-94e2-d64efff7d9ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58\" (UID: \"ddcfb701-535b-4321-94e2-d64efff7d9ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"
Oct 31 01:33:50.453405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.453381    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcfb701-535b-4321-94e2-d64efff7d9ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58\" (UID: \"ddcfb701-535b-4321-94e2-d64efff7d9ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"
Oct 31 01:33:50.466586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.466540    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"
Oct 31 01:33:50.509025 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.508987    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp7zp\" (UniqueName: \"kubernetes.io/projected/ddcfb701-535b-4321-94e2-d64efff7d9ef-kube-api-access-pp7zp\") pod \"payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58\" (UID: \"ddcfb701-535b-4321-94e2-d64efff7d9ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"
Oct 31 01:33:50.556209 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.556143    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:50.586908 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.586813    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"
Oct 31 01:33:50.842009 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.841983    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"]
Oct 31 01:33:50.843778 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:33:50.843750    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396b763c_807d_4f81_9bbc_20ba4696fee4.slice/crio-ab897da71c5e129e9437d51f17896b8167664aad24aca76ee079078fdea52a2e WatchSource:0}: Error finding container ab897da71c5e129e9437d51f17896b8167664aad24aca76ee079078fdea52a2e: Status 404 returned error can't find the container with id ab897da71c5e129e9437d51f17896b8167664aad24aca76ee079078fdea52a2e
Oct 31 01:33:50.967088 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:50.967056    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"]
Oct 31 01:33:50.968215 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:33:50.968149    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddcfb701_535b_4321_94e2_d64efff7d9ef.slice/crio-394baafb165a43b41a7166d6a509ff17de3d6315e81e688e58fca78148fb1f51 WatchSource:0}: Error finding container 394baafb165a43b41a7166d6a509ff17de3d6315e81e688e58fca78148fb1f51: Status 404 returned error can't find the container with id 394baafb165a43b41a7166d6a509ff17de3d6315e81e688e58fca78148fb1f51
Oct 31 01:33:51.020147 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:51.020108    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2" event={"ID":"396b763c-807d-4f81-9bbc-20ba4696fee4","Type":"ContainerStarted","Data":"281adc4625539d87afc32a363b73808ca9691320747a6db3700a6b3bb6e134b5"}
Oct 31 01:33:51.020322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:51.020169    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2" event={"ID":"396b763c-807d-4f81-9bbc-20ba4696fee4","Type":"ContainerStarted","Data":"ab897da71c5e129e9437d51f17896b8167664aad24aca76ee079078fdea52a2e"}
Oct 31 01:33:51.020387 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:51.020316    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"
Oct 31 01:33:51.021463 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:51.021258    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58" event={"ID":"ddcfb701-535b-4321-94e2-d64efff7d9ef","Type":"ContainerStarted","Data":"394baafb165a43b41a7166d6a509ff17de3d6315e81e688e58fca78148fb1f51"}
Oct 31 01:33:51.133658 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:51.133556    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2" podStartSLOduration=1.133542599 podStartE2EDuration="1.133542599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:51.130785305 +0000 UTC m=+2240.820668433" watchObservedRunningTime="2026-10-31 01:33:51.133542599 +0000 UTC m=+2240.823425726"
Oct 31 01:33:51.850527 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:51.850493    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:52.029964 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.029925    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58" event={"ID":"ddcfb701-535b-4321-94e2-d64efff7d9ef","Type":"ContainerStarted","Data":"55681d328992489406079f5eb4bad445b4d2d4b518a4f08bbaa8176a07256375"}
Oct 31 01:33:52.030159 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.030028    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"
Oct 31 01:33:52.143436 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.143337    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58" podStartSLOduration=2.1433228460000002 podStartE2EDuration="2.143322846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:52.142834308 +0000 UTC m=+2241.832717453" watchObservedRunningTime="2026-10-31 01:33:52.143322846 +0000 UTC m=+2241.833205973"
Oct 31 01:33:52.315540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.315512    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"]
Oct 31 01:33:52.319318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.319302    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.399280 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.396230    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-envoy\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.400937 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.400838    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-634135-openshift-default-dockercfg-vftrn\""
Oct 31 01:33:52.401067 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.400913    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-workload-certs\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.401254 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.401232    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-data\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.401540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.401517    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-podinfo\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.401721 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.401704    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-workload-socket\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.401914 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.401897    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsk72\" (UniqueName: \"kubernetes.io/projected/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-kube-api-access-fsk72\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.402108 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.402093    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istiod-ca-cert\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.402358 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.402311    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-credential-socket\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.402538 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.402521    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-token\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.431309 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.431280    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"]
Oct 31 01:33:52.455057 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.455000    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:52.503549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.503514    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-podinfo\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.503734 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.503557    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-workload-socket\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.503734 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.503602    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsk72\" (UniqueName: \"kubernetes.io/projected/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-kube-api-access-fsk72\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.503734 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.503649    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istiod-ca-cert\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.503905 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.503776    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-credential-socket\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.503905 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.503806    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-token\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.503905 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.503882    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-envoy\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.504070 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.503930    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-workload-certs\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.504070 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.503970    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-data\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.504170 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.504081    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-workload-socket\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.504462 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.504444    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istiod-ca-cert\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.504677 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.504656    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-data\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.504797 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.504716    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-credential-socket\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.504797 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.504727    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-workload-certs\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.506339 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.506320    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-podinfo\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.507314 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.507295    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-envoy\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.560205 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.560152    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-token\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.560352 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.560257    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsk72\" (UniqueName: \"kubernetes.io/projected/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-kube-api-access-fsk72\") pod \"e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") " pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.632749 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.632722    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:52.672711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:52.672639    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:53.060001 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:53.059962    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:53.062050 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:53.062026    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"]
Oct 31 01:33:53.063980 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:33:53.063957    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa12d5ea_cb6d_4c40_9f13_ed3c0b717bfa.slice/crio-03cb53e0ff4e926704482e35930a996de4f3cb9057344b24a5b1be180a0df991 WatchSource:0}: Error finding container 03cb53e0ff4e926704482e35930a996de4f3cb9057344b24a5b1be180a0df991: Status 404 returned error can't find the container with id 03cb53e0ff4e926704482e35930a996de4f3cb9057344b24a5b1be180a0df991
Oct 31 01:33:53.068450 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:53.067268    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:33:53.068450 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:53.067358    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:33:53.068450 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:53.067402    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:33:53.241418 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:53.241370    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:53.417743 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:53.417662    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:54.044254 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:54.044212    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8" event={"ID":"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa","Type":"ContainerStarted","Data":"27663cef3d053c2d5799ff5158d38f355fad2f7258eca875cca776395de2d772"}
Oct 31 01:33:54.044468 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:54.044289    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8" event={"ID":"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa","Type":"ContainerStarted","Data":"03cb53e0ff4e926704482e35930a996de4f3cb9057344b24a5b1be180a0df991"}
Oct 31 01:33:54.164605 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:54.164553    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8" podStartSLOduration=2.164539164 podStartE2EDuration="2.164539164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:54.161193751 +0000 UTC m=+2243.851076869" watchObservedRunningTime="2026-10-31 01:33:54.164539164 +0000 UTC m=+2243.854422291"
Oct 31 01:33:54.632871 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:54.632840    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:54.638168 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:54.638147    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:55.050589 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:55.050558    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:55.051618 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:55.051598    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:33:56.975949 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:56.975919    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:57.175482 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:57.175450    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:58.191645 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:33:58.191615    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:59.181116 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:33:59.181084    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:34:01.176830 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.176781    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:01.377010 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.376979    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:01.551795 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.551764    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"]
Oct 31 01:34:01.555757 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.555740    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"
Oct 31 01:34:01.605242 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.605215    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k\" (UID: \"c664aa20-87b7-4ed6-8b7e-d86a5dca7f92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"
Oct 31 01:34:01.605366 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.605332    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvrd\" (UniqueName: \"kubernetes.io/projected/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92-kube-api-access-whvrd\") pod \"payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k\" (UID: \"c664aa20-87b7-4ed6-8b7e-d86a5dca7f92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"
Oct 31 01:34:01.612391 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.612372    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-634135\""
Oct 31 01:34:01.665695 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.665643    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-634135-dockercfg-dwwzm\""
Oct 31 01:34:01.668383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.668329    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"]
Oct 31 01:34:01.706970 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.706932    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k\" (UID: \"c664aa20-87b7-4ed6-8b7e-d86a5dca7f92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"
Oct 31 01:34:01.707133 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.707016    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whvrd\" (UniqueName: \"kubernetes.io/projected/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92-kube-api-access-whvrd\") pod \"payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k\" (UID: \"c664aa20-87b7-4ed6-8b7e-d86a5dca7f92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"
Oct 31 01:34:01.707697 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.707677    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k\" (UID: \"c664aa20-87b7-4ed6-8b7e-d86a5dca7f92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"
Oct 31 01:34:01.714929 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.714910    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"]
Oct 31 01:34:01.718895 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.718881    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"
Oct 31 01:34:01.765931 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.765906    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"]
Oct 31 01:34:01.767951 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.767928    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvrd\" (UniqueName: \"kubernetes.io/projected/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92-kube-api-access-whvrd\") pod \"payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k\" (UID: \"c664aa20-87b7-4ed6-8b7e-d86a5dca7f92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"
Oct 31 01:34:01.808161 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.808089    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f25d63-5370-41c8-a8cd-5843ea7e11f6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4\" (UID: \"c2f25d63-5370-41c8-a8cd-5843ea7e11f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"
Oct 31 01:34:01.808161 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.808130    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fblmv\" (UniqueName: \"kubernetes.io/projected/c2f25d63-5370-41c8-a8cd-5843ea7e11f6-kube-api-access-fblmv\") pod \"payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4\" (UID: \"c2f25d63-5370-41c8-a8cd-5843ea7e11f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"
Oct 31 01:34:01.869152 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.869130    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"
Oct 31 01:34:01.909899 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.909858    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f25d63-5370-41c8-a8cd-5843ea7e11f6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4\" (UID: \"c2f25d63-5370-41c8-a8cd-5843ea7e11f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"
Oct 31 01:34:01.910579 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.910551    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fblmv\" (UniqueName: \"kubernetes.io/projected/c2f25d63-5370-41c8-a8cd-5843ea7e11f6-kube-api-access-fblmv\") pod \"payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4\" (UID: \"c2f25d63-5370-41c8-a8cd-5843ea7e11f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"
Oct 31 01:34:01.910579 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.910572    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f25d63-5370-41c8-a8cd-5843ea7e11f6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4\" (UID: \"c2f25d63-5370-41c8-a8cd-5843ea7e11f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"
Oct 31 01:34:01.967860 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:01.967831    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fblmv\" (UniqueName: \"kubernetes.io/projected/c2f25d63-5370-41c8-a8cd-5843ea7e11f6-kube-api-access-fblmv\") pod \"payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4\" (UID: \"c2f25d63-5370-41c8-a8cd-5843ea7e11f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"
Oct 31 01:34:02.031904 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:02.031874    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"
Oct 31 01:34:02.032882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:02.032864    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"
Oct 31 01:34:02.263671 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:02.263640    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"]
Oct 31 01:34:02.264872 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:34:02.264847    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc664aa20_87b7_4ed6_8b7e_d86a5dca7f92.slice/crio-164ec3792e855655b19c5ef2f8aab4bb723b10c19cc9e20eca3d99d835ff2609 WatchSource:0}: Error finding container 164ec3792e855655b19c5ef2f8aab4bb723b10c19cc9e20eca3d99d835ff2609: Status 404 returned error can't find the container with id 164ec3792e855655b19c5ef2f8aab4bb723b10c19cc9e20eca3d99d835ff2609
Oct 31 01:34:02.409288 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:02.409263    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"]
Oct 31 01:34:02.410509 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:34:02.410477    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f25d63_5370_41c8_a8cd_5843ea7e11f6.slice/crio-cd98f320788ec598a86f84c803123678b2376033c4702f4f5262944aa85a1d5d WatchSource:0}: Error finding container cd98f320788ec598a86f84c803123678b2376033c4702f4f5262944aa85a1d5d: Status 404 returned error can't find the container with id cd98f320788ec598a86f84c803123678b2376033c4702f4f5262944aa85a1d5d
Oct 31 01:34:03.039910 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:03.039878    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"
Oct 31 01:34:03.108220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:03.108165    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k" event={"ID":"c664aa20-87b7-4ed6-8b7e-d86a5dca7f92","Type":"ContainerStarted","Data":"47c74f6f1ed596e89f2a2054dc41afdb4581e48974dcfe9afb3d07f282e2b4ec"}
Oct 31 01:34:03.108379 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:03.108263    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k" event={"ID":"c664aa20-87b7-4ed6-8b7e-d86a5dca7f92","Type":"ContainerStarted","Data":"164ec3792e855655b19c5ef2f8aab4bb723b10c19cc9e20eca3d99d835ff2609"}
Oct 31 01:34:03.108379 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:03.108294    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"
Oct 31 01:34:03.110021 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:03.109987    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4" event={"ID":"c2f25d63-5370-41c8-a8cd-5843ea7e11f6","Type":"ContainerStarted","Data":"df78a93f56fb14e6d09f010f11958ccf2108a42647fb136fffdf260b283b4b4b"}
Oct 31 01:34:03.110159 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:03.110034    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4" event={"ID":"c2f25d63-5370-41c8-a8cd-5843ea7e11f6","Type":"ContainerStarted","Data":"cd98f320788ec598a86f84c803123678b2376033c4702f4f5262944aa85a1d5d"}
Oct 31 01:34:03.110159 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:03.110061    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"
Oct 31 01:34:03.275395 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:03.275341    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k" podStartSLOduration=2.275327456 podStartE2EDuration="2.275327456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:03.272593861 +0000 UTC m=+2252.962476988" watchObservedRunningTime="2026-10-31 01:34:03.275327456 +0000 UTC m=+2252.965210621"
Oct 31 01:34:03.309192 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:03.309111    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:03.387309 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:03.387258    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4" podStartSLOduration=2.387243906 podStartE2EDuration="2.387243906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:03.386028053 +0000 UTC m=+2253.075911180" watchObservedRunningTime="2026-10-31 01:34:03.387243906 +0000 UTC m=+2253.077127033"
Oct 31 01:34:08.331638 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:08.331592    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:13.226556 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:13.226505    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:14.119253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:14.119228    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"
Oct 31 01:34:14.119433 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:14.119292    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"
Oct 31 01:34:16.742270 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:16.742236    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:17.235598 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:17.235563    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:19.223987 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:19.223911    4476 generic.go:356] "Generic (PLEG): container finished" podID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerID="982d584be2c8f70db2d53a525d56dae4c1a02640f0084d0445ecdee832916e4e" exitCode=0
Oct 31 01:34:19.224444 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:19.223980    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv" event={"ID":"185b3025-1c9e-4c04-9beb-a21c93225c0f","Type":"ContainerDied","Data":"982d584be2c8f70db2d53a525d56dae4c1a02640f0084d0445ecdee832916e4e"}
Oct 31 01:34:19.224444 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:19.224386    4476 scope.go:122] "RemoveContainer" containerID="982d584be2c8f70db2d53a525d56dae4c1a02640f0084d0445ecdee832916e4e"
Oct 31 01:34:19.225899 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:19.225872    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:34:19.225966 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:19.225939    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:34:19.226003 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:19.225973    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:34:19.656518 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:19.656484    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:19.836206 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:19.836149    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:20.234381 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:20.234347    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv" event={"ID":"185b3025-1c9e-4c04-9beb-a21c93225c0f","Type":"ContainerStarted","Data":"a6dd4c634b15e3db4891204e05c083111679532de1e106784bf77eb3b3a453a2"}
Oct 31 01:34:21.223218 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:21.223168    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:34:21.223404 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:21.223384    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 01:34:21.223467 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:21.223437    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv" podUID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 01:34:22.223229 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:22.223161    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 01:34:22.223734 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:22.223277    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv" podUID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 01:34:23.223320 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:23.223287    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 01:34:23.223717 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:23.223335    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv" podUID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 01:34:24.223478 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:24.223442    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 01:34:24.223901 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:24.223494    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv" podUID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 01:34:24.531731 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:24.531703    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:25.223269 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:25.223232    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 01:34:25.223463 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:25.223290    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv" podUID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 01:34:26.227575 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:26.227547    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:34:26.228025 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:26.228005    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:34:26.228752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:26.228733    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:34:26.313103 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:26.313078    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:27.045252 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:27.045219    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:28.160888 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:28.160862    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:29.181736 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:34:29.181697    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:34:29.297108 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:29.297079    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:34.179120 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:34.179087    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:36.474290 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:36.474261    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:37.389504 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:37.389470    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:37.658512 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:37.658433    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:37.997074 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:37.996996    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"]
Oct 31 01:34:38.299667 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:38.299586    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:38.497642 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:38.497612    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:40.936977 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:40.936864    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:41.303014 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:41.302987    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:42.997724 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:42.997689    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf" podUID="ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f" containerName="payload-pre-processing" containerID="cri-o://e0c3a26f4aad15bbe4772cb3aa801c3079dbe1789e78a048552a31c807173f61" gracePeriod=25
Oct 31 01:34:43.260455 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.260394    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"
Oct 31 01:34:43.317761 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.317731    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f-plugins-config-volume\") pod \"ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f\" (UID: \"ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f\") "
Oct 31 01:34:43.317905 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.317791    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f-kube-api-access-mcf79\" (UniqueName: \"kubernetes.io/projected/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f-kube-api-access-mcf79\") pod \"ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f\" (UID: \"ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f\") "
Oct 31 01:34:43.318156 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.318126    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f-plugins-config-volume" pod "ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f" (UID: "ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:43.318330 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.318312    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:43.319936 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.319916    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f-kube-api-access-mcf79" pod "ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f" (UID: "ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f"). InnerVolumeSpecName "kube-api-access-mcf79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:43.405436 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.405401    4476 generic.go:356] "Generic (PLEG): container finished" podID="ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f" containerID="e0c3a26f4aad15bbe4772cb3aa801c3079dbe1789e78a048552a31c807173f61" exitCode=0
Oct 31 01:34:43.405626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.405448    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"
Oct 31 01:34:43.405626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.405488    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf" event={"ID":"ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f","Type":"ContainerDied","Data":"e0c3a26f4aad15bbe4772cb3aa801c3079dbe1789e78a048552a31c807173f61"}
Oct 31 01:34:43.405626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.405558    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf" event={"ID":"ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f","Type":"ContainerDied","Data":"6e1e6a63e7078bc74101ea84dedc3348c8e4203e4b3a84fcbef56d1c0db3681a"}
Oct 31 01:34:43.405626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.405593    4476 scope.go:122] "RemoveContainer" containerID="e0c3a26f4aad15bbe4772cb3aa801c3079dbe1789e78a048552a31c807173f61"
Oct 31 01:34:43.415621 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.415607    4476 scope.go:122] "RemoveContainer" containerID="e0c3a26f4aad15bbe4772cb3aa801c3079dbe1789e78a048552a31c807173f61"
Oct 31 01:34:43.415882 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:34:43.415866    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c3a26f4aad15bbe4772cb3aa801c3079dbe1789e78a048552a31c807173f61\": container with ID starting with e0c3a26f4aad15bbe4772cb3aa801c3079dbe1789e78a048552a31c807173f61 not found: ID does not exist" containerID="e0c3a26f4aad15bbe4772cb3aa801c3079dbe1789e78a048552a31c807173f61"
Oct 31 01:34:43.415921 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.415889    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c3a26f4aad15bbe4772cb3aa801c3079dbe1789e78a048552a31c807173f61"} err="failed to get container status \"e0c3a26f4aad15bbe4772cb3aa801c3079dbe1789e78a048552a31c807173f61\": rpc error: code = NotFound desc = could not find container \"e0c3a26f4aad15bbe4772cb3aa801c3079dbe1789e78a048552a31c807173f61\": container with ID starting with e0c3a26f4aad15bbe4772cb3aa801c3079dbe1789e78a048552a31c807173f61 not found: ID does not exist"
Oct 31 01:34:43.419762 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.419741    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcf79\" (UniqueName: \"kubernetes.io/projected/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f-kube-api-access-mcf79\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:43.568653 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.568622    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"]
Oct 31 01:34:43.571237 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:43.571211    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9da8f-78f759f779-s5mpf"]
Oct 31 01:34:44.453849 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:44.453819    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f" path="/var/lib/kubelet/pods/ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f/volumes"
Oct 31 01:34:45.726242 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:45.726196    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:47.899579 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:47.899545    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"]
Oct 31 01:34:47.900098 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:47.900083    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f" containerName="payload-pre-processing"
Oct 31 01:34:47.900142 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:47.900100    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f" containerName="payload-pre-processing"
Oct 31 01:34:47.900228 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:47.900216    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4170e4-5efa-456d-b6f4-a16d2f2d1f9f" containerName="payload-pre-processing"
Oct 31 01:34:47.904897 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:47.904882    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:47.961161 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:47.961130    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a31828-openshift-default-dockercfg-n2c9x\""
Oct 31 01:34:48.014647 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.014613    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"]
Oct 31 01:34:48.051855 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.051820    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:48.071946 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.071913    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-istio-data\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.072130 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.071977    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-istio-envoy\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.072130 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.072030    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w95v6\" (UniqueName: \"kubernetes.io/projected/a535f499-71aa-4da2-823a-16d687bd5d92-kube-api-access-w95v6\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.072279 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.072130    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a535f499-71aa-4da2-823a-16d687bd5d92-istiod-ca-cert\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.072279 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.072162    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-workload-socket\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.072548 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.072310    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-workload-certs\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.072548 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.072343    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a535f499-71aa-4da2-823a-16d687bd5d92-istio-podinfo\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.072548 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.072423    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a535f499-71aa-4da2-823a-16d687bd5d92-istio-token\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.072548 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.072455    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-credential-socket\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.173283 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.173212    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-istio-envoy\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.173283 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.173247    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w95v6\" (UniqueName: \"kubernetes.io/projected/a535f499-71aa-4da2-823a-16d687bd5d92-kube-api-access-w95v6\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.173283 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.173277    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a535f499-71aa-4da2-823a-16d687bd5d92-istiod-ca-cert\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.173549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.173298    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-workload-socket\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.173549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.173403    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-workload-certs\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.173549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.173445    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a535f499-71aa-4da2-823a-16d687bd5d92-istio-podinfo\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.173549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.173518    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a535f499-71aa-4da2-823a-16d687bd5d92-istio-token\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.173779 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.173557    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-credential-socket\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.173779 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.173630    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-istio-data\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.173779 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.173769    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-workload-socket\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.173970 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.173938    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-workload-certs\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.174029 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.173987    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-istio-data\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.174029 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.174017    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-credential-socket\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.174098 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.174055    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a535f499-71aa-4da2-823a-16d687bd5d92-istiod-ca-cert\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.175644 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.175625    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-istio-envoy\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.176152 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.176091    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a535f499-71aa-4da2-823a-16d687bd5d92-istio-podinfo\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.230272 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.230251    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w95v6\" (UniqueName: \"kubernetes.io/projected/a535f499-71aa-4da2-823a-16d687bd5d92-kube-api-access-w95v6\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.230405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.230387    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a535f499-71aa-4da2-823a-16d687bd5d92-istio-token\") pod \"e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") " pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.359199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.359132    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:48.518057 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.517976    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:48.609073 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.609043    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:48.946879 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.946853    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"]
Oct 31 01:34:48.947701 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:34:48.947674    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda535f499_71aa_4da2_823a_16d687bd5d92.slice/crio-ace828b58647d04902c2cb8a627609ab84cacb8c55d6957f20bec9a71436ed45 WatchSource:0}: Error finding container ace828b58647d04902c2cb8a627609ab84cacb8c55d6957f20bec9a71436ed45: Status 404 returned error can't find the container with id ace828b58647d04902c2cb8a627609ab84cacb8c55d6957f20bec9a71436ed45
Oct 31 01:34:48.950222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.950165    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:34:48.950328 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.950271    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:34:48.950328 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:48.950313    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:34:49.441890 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:49.441851    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9" event={"ID":"a535f499-71aa-4da2-823a-16d687bd5d92","Type":"ContainerStarted","Data":"c4d60dffbc9a377a159ec1828d47e4fa900d14243a1c58a247506fd48361c480"}
Oct 31 01:34:49.442091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:49.441903    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9" event={"ID":"a535f499-71aa-4da2-823a-16d687bd5d92","Type":"ContainerStarted","Data":"ace828b58647d04902c2cb8a627609ab84cacb8c55d6957f20bec9a71436ed45"}
Oct 31 01:34:49.518225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:49.518196    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:49.561344 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:49.561299    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9" podStartSLOduration=2.561285359 podStartE2EDuration="2.561285359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:49.560114975 +0000 UTC m=+2299.249998135" watchObservedRunningTime="2026-10-31 01:34:49.561285359 +0000 UTC m=+2299.251168561"
Oct 31 01:34:50.472368 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:50.472335    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:50.523870 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:50.523836    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:50.664062 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:50.664028    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:50.901799 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:50.901769    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:51.454418 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:51.454385    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:51.455457 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:51.455437    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:34:52.530407 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:52.530379    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:52.740729 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:52.740698    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:55.703454 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:55.703420    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:55.901266 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:55.901234    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:56.079718 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:56.079682    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"]
Oct 31 01:34:56.083626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:56.083605    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"
Oct 31 01:34:56.093210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:56.093166    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:56.137893 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:56.137867    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a31828\""
Oct 31 01:34:56.197650 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:56.197619    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a31828-dockercfg-vng7w\""
Oct 31 01:34:56.207311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:56.207268    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"]
Oct 31 01:34:56.207822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:56.207786    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1140ac0a-116a-4f86-9f5d-803e1f3a4667-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq\" (UID: \"1140ac0a-116a-4f86-9f5d-803e1f3a4667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"
Oct 31 01:34:56.207939 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:56.207826    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb6dn\" (UniqueName: \"kubernetes.io/projected/1140ac0a-116a-4f86-9f5d-803e1f3a4667-kube-api-access-zb6dn\") pod \"payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq\" (UID: \"1140ac0a-116a-4f86-9f5d-803e1f3a4667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"
Oct 31 01:34:56.312993 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:56.312951    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1140ac0a-116a-4f86-9f5d-803e1f3a4667-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq\" (UID: \"1140ac0a-116a-4f86-9f5d-803e1f3a4667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"
Oct 31 01:34:56.319331 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:56.319277    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb6dn\" (UniqueName: \"kubernetes.io/projected/1140ac0a-116a-4f86-9f5d-803e1f3a4667-kube-api-access-zb6dn\") pod \"payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq\" (UID: \"1140ac0a-116a-4f86-9f5d-803e1f3a4667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"
Oct 31 01:34:56.319561 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:56.319531    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1140ac0a-116a-4f86-9f5d-803e1f3a4667-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq\" (UID: \"1140ac0a-116a-4f86-9f5d-803e1f3a4667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"
Oct 31 01:34:56.376752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:56.376681    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb6dn\" (UniqueName: \"kubernetes.io/projected/1140ac0a-116a-4f86-9f5d-803e1f3a4667-kube-api-access-zb6dn\") pod \"payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq\" (UID: \"1140ac0a-116a-4f86-9f5d-803e1f3a4667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"
Oct 31 01:34:56.397729 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:56.397706    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"
Oct 31 01:34:56.780327 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:56.780304    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"]
Oct 31 01:34:56.782459 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:34:56.782437    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1140ac0a_116a_4f86_9f5d_803e1f3a4667.slice/crio-222e4411b0d009f06bdea3fcc4c7f34b2b7ecbf23872a12d844b4ec097c6d74a WatchSource:0}: Error finding container 222e4411b0d009f06bdea3fcc4c7f34b2b7ecbf23872a12d844b4ec097c6d74a: Status 404 returned error can't find the container with id 222e4411b0d009f06bdea3fcc4c7f34b2b7ecbf23872a12d844b4ec097c6d74a
Oct 31 01:34:57.490232 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:57.490201    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq" event={"ID":"1140ac0a-116a-4f86-9f5d-803e1f3a4667","Type":"ContainerStarted","Data":"727c36125b8f64d53b1931de4ea05d65246c3e33466641c6b8d819a1406d5d8d"}
Oct 31 01:34:57.490412 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:57.490250    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq" event={"ID":"1140ac0a-116a-4f86-9f5d-803e1f3a4667","Type":"ContainerStarted","Data":"222e4411b0d009f06bdea3fcc4c7f34b2b7ecbf23872a12d844b4ec097c6d74a"}
Oct 31 01:34:57.490412 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:57.490281    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"
Oct 31 01:34:57.609253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:34:57.609204    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq" podStartSLOduration=1.609190838 podStartE2EDuration="1.609190838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:57.608208948 +0000 UTC m=+2307.298092072" watchObservedRunningTime="2026-10-31 01:34:57.609190838 +0000 UTC m=+2307.299073955"
Oct 31 01:34:59.180628 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:34:59.180589    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:35:02.489931 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:02.489897    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:07.113375 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:07.113330    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:07.314882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:07.314851    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:07.559899 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:07.559870    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:08.499327 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:08.499301    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"
Oct 31 01:35:12.097240 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:12.097209    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:12.676475 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:12.676420    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:13.001224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:13.001129    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:13.854626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:13.854589    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:14.055058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:14.055029    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:17.560841 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:17.560809    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:20.955099 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:20.955068    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:21.128202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:21.128157    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:22.297501 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:22.297457    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:25.379800 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:25.379766    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"]
Oct 31 01:35:25.380245 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:25.379996    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl" podUID="66b1c85a-f9d0-4502-b529-29df7bc5b837" containerName="istio-proxy" containerID="cri-o://ef22bc96c4c21871dd870b18e9cae8c78a42d4b0676b588c1a9bedf17680931b" gracePeriod=30
Oct 31 01:35:26.532684 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:26.532647    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-9ngbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:35:26.533084 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:26.532719    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl" podUID="66b1c85a-f9d0-4502-b529-29df7bc5b837" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:35:27.209299 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:27.209267    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:28.638890 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:28.638860    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:28.852484 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:28.852450    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:29.181329 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:35:29.181291    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:35:30.647660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.647590    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:35:30.683835 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.683799    4476 generic.go:356] "Generic (PLEG): container finished" podID="66b1c85a-f9d0-4502-b529-29df7bc5b837" containerID="ef22bc96c4c21871dd870b18e9cae8c78a42d4b0676b588c1a9bedf17680931b" exitCode=0
Oct 31 01:35:30.684000 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.683851    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"
Oct 31 01:35:30.684000 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.683883    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl" event={"ID":"66b1c85a-f9d0-4502-b529-29df7bc5b837","Type":"ContainerDied","Data":"ef22bc96c4c21871dd870b18e9cae8c78a42d4b0676b588c1a9bedf17680931b"}
Oct 31 01:35:30.684000 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.683944    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl" event={"ID":"66b1c85a-f9d0-4502-b529-29df7bc5b837","Type":"ContainerDied","Data":"49257d2076758c071a3c19d9a1b8deec4ea0822e75fb31cfefda88d2c63f5e1a"}
Oct 31 01:35:30.684000 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.683963    4476 scope.go:122] "RemoveContainer" containerID="ef22bc96c4c21871dd870b18e9cae8c78a42d4b0676b588c1a9bedf17680931b"
Oct 31 01:35:30.693519 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.693493    4476 scope.go:122] "RemoveContainer" containerID="ef22bc96c4c21871dd870b18e9cae8c78a42d4b0676b588c1a9bedf17680931b"
Oct 31 01:35:30.693810 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:35:30.693792    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef22bc96c4c21871dd870b18e9cae8c78a42d4b0676b588c1a9bedf17680931b\": container with ID starting with ef22bc96c4c21871dd870b18e9cae8c78a42d4b0676b588c1a9bedf17680931b not found: ID does not exist" containerID="ef22bc96c4c21871dd870b18e9cae8c78a42d4b0676b588c1a9bedf17680931b"
Oct 31 01:35:30.693864 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.693824    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef22bc96c4c21871dd870b18e9cae8c78a42d4b0676b588c1a9bedf17680931b"} err="failed to get container status \"ef22bc96c4c21871dd870b18e9cae8c78a42d4b0676b588c1a9bedf17680931b\": rpc error: code = NotFound desc = could not find container \"ef22bc96c4c21871dd870b18e9cae8c78a42d4b0676b588c1a9bedf17680931b\": container with ID starting with ef22bc96c4c21871dd870b18e9cae8c78a42d4b0676b588c1a9bedf17680931b not found: ID does not exist"
Oct 31 01:35:30.812376 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.812340    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-envoy\") pod \"66b1c85a-f9d0-4502-b529-29df7bc5b837\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") "
Oct 31 01:35:30.812563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.812438    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-workload-certs\") pod \"66b1c85a-f9d0-4502-b529-29df7bc5b837\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") "
Oct 31 01:35:30.812563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.812484    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-data\") pod \"66b1c85a-f9d0-4502-b529-29df7bc5b837\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") "
Oct 31 01:35:30.812563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.812514    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b1c85a-f9d0-4502-b529-29df7bc5b837-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b1c85a-f9d0-4502-b529-29df7bc5b837-istiod-ca-cert\") pod \"66b1c85a-f9d0-4502-b529-29df7bc5b837\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") "
Oct 31 01:35:30.812563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.812544    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-token\" (UniqueName: \"kubernetes.io/projected/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-token\") pod \"66b1c85a-f9d0-4502-b529-29df7bc5b837\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") "
Oct 31 01:35:30.812791 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.812590    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-workload-socket\") pod \"66b1c85a-f9d0-4502-b529-29df7bc5b837\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") "
Oct 31 01:35:30.812791 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.812621    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b1c85a-f9d0-4502-b529-29df7bc5b837-kube-api-access-pnz5m\" (UniqueName: \"kubernetes.io/projected/66b1c85a-f9d0-4502-b529-29df7bc5b837-kube-api-access-pnz5m\") pod \"66b1c85a-f9d0-4502-b529-29df7bc5b837\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") "
Oct 31 01:35:30.812791 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.812665    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-podinfo\") pod \"66b1c85a-f9d0-4502-b529-29df7bc5b837\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") "
Oct 31 01:35:30.812791 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.812700    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-credential-socket\") pod \"66b1c85a-f9d0-4502-b529-29df7bc5b837\" (UID: \"66b1c85a-f9d0-4502-b529-29df7bc5b837\") "
Oct 31 01:35:30.813007 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.812871    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-workload-socket" pod "66b1c85a-f9d0-4502-b529-29df7bc5b837" (UID: "66b1c85a-f9d0-4502-b529-29df7bc5b837"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:30.813007 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.812966    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b1c85a-f9d0-4502-b529-29df7bc5b837-istiod-ca-cert" pod "66b1c85a-f9d0-4502-b529-29df7bc5b837" (UID: "66b1c85a-f9d0-4502-b529-29df7bc5b837"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:35:30.813113 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.813002    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-workload-certs" pod "66b1c85a-f9d0-4502-b529-29df7bc5b837" (UID: "66b1c85a-f9d0-4502-b529-29df7bc5b837"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:30.813113 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.813072    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-data" pod "66b1c85a-f9d0-4502-b529-29df7bc5b837" (UID: "66b1c85a-f9d0-4502-b529-29df7bc5b837"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:30.813113 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.813089    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-credential-socket" pod "66b1c85a-f9d0-4502-b529-29df7bc5b837" (UID: "66b1c85a-f9d0-4502-b529-29df7bc5b837"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:30.813314 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.813294    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:30.813405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.813320    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:30.813405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.813335    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b1c85a-f9d0-4502-b529-29df7bc5b837-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:30.813405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.813349    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:30.813405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.813362    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:30.815132 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.815102    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-podinfo" pod "66b1c85a-f9d0-4502-b529-29df7bc5b837" (UID: "66b1c85a-f9d0-4502-b529-29df7bc5b837"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:35:30.815244 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.815137    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-token" pod "66b1c85a-f9d0-4502-b529-29df7bc5b837" (UID: "66b1c85a-f9d0-4502-b529-29df7bc5b837"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:30.815411 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.815394    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-envoy" pod "66b1c85a-f9d0-4502-b529-29df7bc5b837" (UID: "66b1c85a-f9d0-4502-b529-29df7bc5b837"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:30.815471 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.815452    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b1c85a-f9d0-4502-b529-29df7bc5b837-kube-api-access-pnz5m" pod "66b1c85a-f9d0-4502-b529-29df7bc5b837" (UID: "66b1c85a-f9d0-4502-b529-29df7bc5b837"). InnerVolumeSpecName "kube-api-access-pnz5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:30.914638 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.914542    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:30.914638 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.914573    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:30.914638 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.914589    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnz5m\" (UniqueName: \"kubernetes.io/projected/66b1c85a-f9d0-4502-b529-29df7bc5b837-kube-api-access-pnz5m\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:30.914638 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:30.914602    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b1c85a-f9d0-4502-b529-29df7bc5b837-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:31.118334 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:31.118301    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"]
Oct 31 01:35:31.217738 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:31.217649    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-9ngbl"]
Oct 31 01:35:31.925438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:31.925402    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:32.121586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.121519    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:32.322541 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.322509    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"]
Oct 31 01:35:32.323109 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.323094    4476 cpu_manager.go:414] "containerMap: removing container" podUID="66b1c85a-f9d0-4502-b529-29df7bc5b837" containerName="istio-proxy"
Oct 31 01:35:32.323167 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.323110    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b1c85a-f9d0-4502-b529-29df7bc5b837" containerName="istio-proxy"
Oct 31 01:35:32.323241 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.323229    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b1c85a-f9d0-4502-b529-29df7bc5b837" containerName="istio-proxy"
Oct 31 01:35:32.327967 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.327949    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"
Oct 31 01:35:32.382165 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.382115    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3f0818\""
Oct 31 01:35:32.385756 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.385727    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"]
Oct 31 01:35:32.430944 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.430903    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8f4f64-5ebd-4331-a385-91dcaed57849-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4\" (UID: \"ff8f4f64-5ebd-4331-a385-91dcaed57849\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"
Oct 31 01:35:32.431438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.431413    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddpgq\" (UniqueName: \"kubernetes.io/projected/ff8f4f64-5ebd-4331-a385-91dcaed57849-kube-api-access-ddpgq\") pod \"payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4\" (UID: \"ff8f4f64-5ebd-4331-a385-91dcaed57849\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"
Oct 31 01:35:32.432087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.431837    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3f0818-dockercfg-8fw6g\""
Oct 31 01:35:32.441467 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.441427    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"]
Oct 31 01:35:32.450803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.450307    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"
Oct 31 01:35:32.462128 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.462094    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b1c85a-f9d0-4502-b529-29df7bc5b837" path="/var/lib/kubelet/pods/66b1c85a-f9d0-4502-b529-29df7bc5b837/volumes"
Oct 31 01:35:32.504265 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.504233    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"]
Oct 31 01:35:32.533616 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.533580    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8f4f64-5ebd-4331-a385-91dcaed57849-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4\" (UID: \"ff8f4f64-5ebd-4331-a385-91dcaed57849\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"
Oct 31 01:35:32.533858 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.533704    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddpgq\" (UniqueName: \"kubernetes.io/projected/ff8f4f64-5ebd-4331-a385-91dcaed57849-kube-api-access-ddpgq\") pod \"payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4\" (UID: \"ff8f4f64-5ebd-4331-a385-91dcaed57849\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"
Oct 31 01:35:32.533858 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.533761    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae91494-86a8-468d-a16d-541b75623712-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj\" (UID: \"aae91494-86a8-468d-a16d-541b75623712\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"
Oct 31 01:35:32.533995 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.533889    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjbgv\" (UniqueName: \"kubernetes.io/projected/aae91494-86a8-468d-a16d-541b75623712-kube-api-access-sjbgv\") pod \"payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj\" (UID: \"aae91494-86a8-468d-a16d-541b75623712\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"
Oct 31 01:35:32.534466 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.534443    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8f4f64-5ebd-4331-a385-91dcaed57849-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4\" (UID: \"ff8f4f64-5ebd-4331-a385-91dcaed57849\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"
Oct 31 01:35:32.595509 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.595436    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddpgq\" (UniqueName: \"kubernetes.io/projected/ff8f4f64-5ebd-4331-a385-91dcaed57849-kube-api-access-ddpgq\") pod \"payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4\" (UID: \"ff8f4f64-5ebd-4331-a385-91dcaed57849\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"
Oct 31 01:35:32.635620 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.635583    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjbgv\" (UniqueName: \"kubernetes.io/projected/aae91494-86a8-468d-a16d-541b75623712-kube-api-access-sjbgv\") pod \"payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj\" (UID: \"aae91494-86a8-468d-a16d-541b75623712\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"
Oct 31 01:35:32.635823 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.635731    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae91494-86a8-468d-a16d-541b75623712-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj\" (UID: \"aae91494-86a8-468d-a16d-541b75623712\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"
Oct 31 01:35:32.636392 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.636372    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae91494-86a8-468d-a16d-541b75623712-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj\" (UID: \"aae91494-86a8-468d-a16d-541b75623712\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"
Oct 31 01:35:32.641489 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.641468    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"
Oct 31 01:35:32.696975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.696945    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjbgv\" (UniqueName: \"kubernetes.io/projected/aae91494-86a8-468d-a16d-541b75623712-kube-api-access-sjbgv\") pod \"payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj\" (UID: \"aae91494-86a8-468d-a16d-541b75623712\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"
Oct 31 01:35:32.776098 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:32.776064    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"
Oct 31 01:35:33.024082 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:33.024050    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"]
Oct 31 01:35:33.025718 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:35:33.025691    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8f4f64_5ebd_4331_a385_91dcaed57849.slice/crio-ad3ed21080566bad31c26c57dcbe79af1f6a6748f320b65db6a99e9dc038a2ed WatchSource:0}: Error finding container ad3ed21080566bad31c26c57dcbe79af1f6a6748f320b65db6a99e9dc038a2ed: Status 404 returned error can't find the container with id ad3ed21080566bad31c26c57dcbe79af1f6a6748f320b65db6a99e9dc038a2ed
Oct 31 01:35:33.160334 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:33.160308    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"]
Oct 31 01:35:33.161584 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:35:33.161542    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae91494_86a8_468d_a16d_541b75623712.slice/crio-39096616371d2282386098a73ba56d148b93ed52084689990d982ac47998fe92 WatchSource:0}: Error finding container 39096616371d2282386098a73ba56d148b93ed52084689990d982ac47998fe92: Status 404 returned error can't find the container with id 39096616371d2282386098a73ba56d148b93ed52084689990d982ac47998fe92
Oct 31 01:35:33.707001 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:33.706963    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj" event={"ID":"aae91494-86a8-468d-a16d-541b75623712","Type":"ContainerStarted","Data":"58ef05694fe0b076863379ae00cb49bfbf04ae9be4f9813483dce30a0d3a8a9d"}
Oct 31 01:35:33.707208 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:33.707027    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj" event={"ID":"aae91494-86a8-468d-a16d-541b75623712","Type":"ContainerStarted","Data":"39096616371d2282386098a73ba56d148b93ed52084689990d982ac47998fe92"}
Oct 31 01:35:33.707208 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:33.707049    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"
Oct 31 01:35:33.708673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:33.708648    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4" event={"ID":"ff8f4f64-5ebd-4331-a385-91dcaed57849","Type":"ContainerStarted","Data":"34d1e7be4f464b86323505c2b864bf1365bb3d1f616550e4826cb4ae22c3315e"}
Oct 31 01:35:33.708808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:33.708686    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4" event={"ID":"ff8f4f64-5ebd-4331-a385-91dcaed57849","Type":"ContainerStarted","Data":"ad3ed21080566bad31c26c57dcbe79af1f6a6748f320b65db6a99e9dc038a2ed"}
Oct 31 01:35:33.708808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:33.708778    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"
Oct 31 01:35:33.821812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:33.821745    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj" podStartSLOduration=1.821725889 podStartE2EDuration="1.821725889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:33.820056265 +0000 UTC m=+2343.509939391" watchObservedRunningTime="2026-10-31 01:35:33.821725889 +0000 UTC m=+2343.511609015"
Oct 31 01:35:34.363871 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:34.363809    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:37.945498 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:37.945463    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:39.594250 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:39.594214    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:43.772323 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:43.772290    4476 generic.go:356] "Generic (PLEG): container finished" podID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" containerID="27663cef3d053c2d5799ff5158d38f355fad2f7258eca875cca776395de2d772" exitCode=0
Oct 31 01:35:43.772712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:43.772368    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8" event={"ID":"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa","Type":"ContainerDied","Data":"27663cef3d053c2d5799ff5158d38f355fad2f7258eca875cca776395de2d772"}
Oct 31 01:35:43.772857 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:43.772840    4476 scope.go:122] "RemoveContainer" containerID="27663cef3d053c2d5799ff5158d38f355fad2f7258eca875cca776395de2d772"
Oct 31 01:35:43.774557 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:43.774524    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:35:43.774666 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:43.774613    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:35:43.774731 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:43.774661    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:35:43.979015 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:43.978926    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4" podStartSLOduration=11.978912332 podStartE2EDuration="11.978912332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:33.939649298 +0000 UTC m=+2343.629532425" watchObservedRunningTime="2026-10-31 01:35:43.978912332 +0000 UTC m=+2353.668795486"
Oct 31 01:35:44.718001 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:44.717971    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"
Oct 31 01:35:44.718201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:44.718037    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"
Oct 31 01:35:44.780666 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:44.780633    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8" event={"ID":"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa","Type":"ContainerStarted","Data":"3aac0e9d4be79aac157683651330fec8738f2ff408137294a5635e9a32793cf2"}
Oct 31 01:35:44.885854 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:44.885825    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:45.633941 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:45.633906    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:35:45.634126 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:45.634040    4476 patch_prober.go:29] interesting pod/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:35:45.634126 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:45.634095    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8" podUID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 01:35:46.633305 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:46.633268    4476 patch_prober.go:29] interesting pod/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:35:46.633775 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:46.633324    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8" podUID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 01:35:47.039549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:47.039516    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:47.319426 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:47.319335    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:47.585493 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:47.585408    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:47.633284 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:47.633254    4476 patch_prober.go:29] interesting pod/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:35:47.633441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:47.633306    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8" podUID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 01:35:48.633572 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:48.633539    4476 patch_prober.go:29] interesting pod/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:35:48.634031 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:48.633603    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8" podUID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 01:35:48.641944 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:48.641923    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:48.884258 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:48.879042    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:49.067202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:49.064661    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:49.633602 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:49.633571    4476 patch_prober.go:29] interesting pod/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:35:49.634108 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:49.633622    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8" podUID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 31 01:35:50.639393 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:50.639364    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:35:50.639898 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:50.639658    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:35:50.640926 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:50.640904    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:35:51.819036 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:51.818991    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:58.253662 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:35:58.253632    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:59.180658 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:35:59.180624    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:36:03.090020 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:03.089982    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:07.930455 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:07.930421    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:12.394113 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.394031    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"]
Oct 31 01:36:12.398308 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.398290    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:12.500589 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.500560    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4hsn\" (UniqueName: \"kubernetes.io/projected/bcf8b55a-0b39-4e7e-9603-70892f505536-kube-api-access-d4hsn\") pod \"maas-api-e2e-ipp-b-a31828-85549b495b-k72q2\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:12.500768 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.500624    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a31828-85549b495b-k72q2\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:12.500768 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.500652    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-metrics-certs\") pod \"maas-api-e2e-ipp-b-a31828-85549b495b-k72q2\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:12.500846 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.500803    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf8b55a-0b39-4e7e-9603-70892f505536-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a31828-85549b495b-k72q2\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:12.501892 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.501874    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a31828-serving-cert\""
Oct 31 01:36:12.504865 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.504846    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"]
Oct 31 01:36:12.601897 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.601863    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4hsn\" (UniqueName: \"kubernetes.io/projected/bcf8b55a-0b39-4e7e-9603-70892f505536-kube-api-access-d4hsn\") pod \"maas-api-e2e-ipp-b-a31828-85549b495b-k72q2\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:12.602091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.601911    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a31828-85549b495b-k72q2\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:12.602091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.601933    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-metrics-certs\") pod \"maas-api-e2e-ipp-b-a31828-85549b495b-k72q2\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:12.602091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.601965    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf8b55a-0b39-4e7e-9603-70892f505536-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a31828-85549b495b-k72q2\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:12.602091 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:36:12.602023    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-serving-cert: secret "maas-api-e2e-ipp-b-a31828-serving-cert" not found
Oct 31 01:36:12.602091 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:36:12.602082    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-maas-api-tls podName:bcf8b55a-0b39-4e7e-9603-70892f505536 nodeName:}" failed. No retries permitted until 2026-10-31 01:36:13.10206701 +0000 UTC m=+2382.791950115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-maas-api-tls") pod "maas-api-e2e-ipp-b-a31828-85549b495b-k72q2" (UID: "bcf8b55a-0b39-4e7e-9603-70892f505536") : secret "maas-api-e2e-ipp-b-a31828-serving-cert" not found
Oct 31 01:36:12.602629 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.602611    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf8b55a-0b39-4e7e-9603-70892f505536-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a31828-85549b495b-k72q2\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:12.604621 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.604597    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-metrics-certs\") pod \"maas-api-e2e-ipp-b-a31828-85549b495b-k72q2\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:12.633864 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.633828    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:12.660612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.660537    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4hsn\" (UniqueName: \"kubernetes.io/projected/bcf8b55a-0b39-4e7e-9603-70892f505536-kube-api-access-d4hsn\") pod \"maas-api-e2e-ipp-b-a31828-85549b495b-k72q2\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:12.830807 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:12.830776    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:13.109595 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:13.109560    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a31828-85549b495b-k72q2\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:13.112163 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:13.112145    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a31828-85549b495b-k72q2\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:13.311014 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:13.310986    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:13.691693 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:13.691662    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"]
Oct 31 01:36:13.692699 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:36:13.692670    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf8b55a_0b39_4e7e_9603_70892f505536.slice/crio-33ef4c5bf64a3f3a3762dae7a77a91160cd30edc4f075859294d33697100f50b WatchSource:0}: Error finding container 33ef4c5bf64a3f3a3762dae7a77a91160cd30edc4f075859294d33697100f50b: Status 404 returned error can't find the container with id 33ef4c5bf64a3f3a3762dae7a77a91160cd30edc4f075859294d33697100f50b
Oct 31 01:36:13.694125 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:13.694106    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:36:13.961506 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:13.961423    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2" event={"ID":"bcf8b55a-0b39-4e7e-9603-70892f505536","Type":"ContainerStarted","Data":"33ef4c5bf64a3f3a3762dae7a77a91160cd30edc4f075859294d33697100f50b"}
Oct 31 01:36:14.969321 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:14.969237    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2" event={"ID":"bcf8b55a-0b39-4e7e-9603-70892f505536","Type":"ContainerStarted","Data":"e6b8165c46c8aa8a5948ecfc9352aec6a7045772fb7f8ed08565bf1badfc5baa"}
Oct 31 01:36:14.969699 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:14.969432    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:14.997491 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:14.997463    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:15.100468 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:15.100424    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2" podStartSLOduration=2.250997556 podStartE2EDuration="3.100410909s" totalImagesPullingTime="849.413353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:15.097910935 +0000 UTC m=+2384.787794062" watchObservedRunningTime="2026-10-31 01:36:15.100410909 +0000 UTC m=+2384.790294035"
Oct 31 01:36:16.800914 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:16.800873    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"]
Oct 31 01:36:16.912831 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:16.912801    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"]
Oct 31 01:36:17.388002 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:17.387965    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:17.633303 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:17.633273    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:18.528196 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:18.528150    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:20.981662 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:20.981636    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:36:21.802075 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:21.802024    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58" podUID="ddcfb701-535b-4321-94e2-d64efff7d9ef" containerName="payload-processing" containerID="cri-o://55681d328992489406079f5eb4bad445b4d2d4b518a4f08bbaa8176a07256375" gracePeriod=25
Oct 31 01:36:21.913894 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:21.913859    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2" podUID="396b763c-807d-4f81-9bbc-20ba4696fee4" containerName="payload-pre-processing" containerID="cri-o://281adc4625539d87afc32a363b73808ca9691320747a6db3700a6b3bb6e134b5" gracePeriod=25
Oct 31 01:36:22.015243 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.015213    4476 generic.go:356] "Generic (PLEG): container finished" podID="ddcfb701-535b-4321-94e2-d64efff7d9ef" containerID="55681d328992489406079f5eb4bad445b4d2d4b518a4f08bbaa8176a07256375" exitCode=0
Oct 31 01:36:22.015683 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.015294    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58" event={"ID":"ddcfb701-535b-4321-94e2-d64efff7d9ef","Type":"ContainerDied","Data":"55681d328992489406079f5eb4bad445b4d2d4b518a4f08bbaa8176a07256375"}
Oct 31 01:36:22.031671 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:36:22.031648    4476 logging.go:55] [core] [Channel #2832 SubChannel #2833]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.102:50052", ServerName: "10.133.0.102:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.102:50052: connect: connection refused"
Oct 31 01:36:22.067725 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.067649    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"
Oct 31 01:36:22.116342 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.116304    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddcfb701-535b-4321-94e2-d64efff7d9ef-kube-api-access-pp7zp\" (UniqueName: \"kubernetes.io/projected/ddcfb701-535b-4321-94e2-d64efff7d9ef-kube-api-access-pp7zp\") pod \"ddcfb701-535b-4321-94e2-d64efff7d9ef\" (UID: \"ddcfb701-535b-4321-94e2-d64efff7d9ef\") "
Oct 31 01:36:22.116558 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.116542    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddcfb701-535b-4321-94e2-d64efff7d9ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcfb701-535b-4321-94e2-d64efff7d9ef-plugins-config-volume\") pod \"ddcfb701-535b-4321-94e2-d64efff7d9ef\" (UID: \"ddcfb701-535b-4321-94e2-d64efff7d9ef\") "
Oct 31 01:36:22.116987 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.116954    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddcfb701-535b-4321-94e2-d64efff7d9ef-plugins-config-volume" pod "ddcfb701-535b-4321-94e2-d64efff7d9ef" (UID: "ddcfb701-535b-4321-94e2-d64efff7d9ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:22.117097 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.117077    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcfb701-535b-4321-94e2-d64efff7d9ef-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:22.118811 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.118772    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddcfb701-535b-4321-94e2-d64efff7d9ef-kube-api-access-pp7zp" pod "ddcfb701-535b-4321-94e2-d64efff7d9ef" (UID: "ddcfb701-535b-4321-94e2-d64efff7d9ef"). InnerVolumeSpecName "kube-api-access-pp7zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:22.162326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.162304    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"
Oct 31 01:36:22.217892 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.217864    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396b763c-807d-4f81-9bbc-20ba4696fee4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396b763c-807d-4f81-9bbc-20ba4696fee4-plugins-config-volume\") pod \"396b763c-807d-4f81-9bbc-20ba4696fee4\" (UID: \"396b763c-807d-4f81-9bbc-20ba4696fee4\") "
Oct 31 01:36:22.218052 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.217976    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396b763c-807d-4f81-9bbc-20ba4696fee4-kube-api-access-726vg\" (UniqueName: \"kubernetes.io/projected/396b763c-807d-4f81-9bbc-20ba4696fee4-kube-api-access-726vg\") pod \"396b763c-807d-4f81-9bbc-20ba4696fee4\" (UID: \"396b763c-807d-4f81-9bbc-20ba4696fee4\") "
Oct 31 01:36:22.218273 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.218250    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396b763c-807d-4f81-9bbc-20ba4696fee4-plugins-config-volume" pod "396b763c-807d-4f81-9bbc-20ba4696fee4" (UID: "396b763c-807d-4f81-9bbc-20ba4696fee4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:22.218448 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.218433    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp7zp\" (UniqueName: \"kubernetes.io/projected/ddcfb701-535b-4321-94e2-d64efff7d9ef-kube-api-access-pp7zp\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:22.218507 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.218455    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396b763c-807d-4f81-9bbc-20ba4696fee4-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:22.220239 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.220214    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396b763c-807d-4f81-9bbc-20ba4696fee4-kube-api-access-726vg" pod "396b763c-807d-4f81-9bbc-20ba4696fee4" (UID: "396b763c-807d-4f81-9bbc-20ba4696fee4"). InnerVolumeSpecName "kube-api-access-726vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:22.320278 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:22.320200    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-726vg\" (UniqueName: \"kubernetes.io/projected/396b763c-807d-4f81-9bbc-20ba4696fee4-kube-api-access-726vg\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:23.023137 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.023101    4476 generic.go:356] "Generic (PLEG): container finished" podID="396b763c-807d-4f81-9bbc-20ba4696fee4" containerID="281adc4625539d87afc32a363b73808ca9691320747a6db3700a6b3bb6e134b5" exitCode=0
Oct 31 01:36:23.023608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.023157    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"
Oct 31 01:36:23.023608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.023202    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2" event={"ID":"396b763c-807d-4f81-9bbc-20ba4696fee4","Type":"ContainerDied","Data":"281adc4625539d87afc32a363b73808ca9691320747a6db3700a6b3bb6e134b5"}
Oct 31 01:36:23.023608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.023285    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2" event={"ID":"396b763c-807d-4f81-9bbc-20ba4696fee4","Type":"ContainerDied","Data":"ab897da71c5e129e9437d51f17896b8167664aad24aca76ee079078fdea52a2e"}
Oct 31 01:36:23.023608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.023313    4476 scope.go:122] "RemoveContainer" containerID="281adc4625539d87afc32a363b73808ca9691320747a6db3700a6b3bb6e134b5"
Oct 31 01:36:23.024942 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.024917    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58" event={"ID":"ddcfb701-535b-4321-94e2-d64efff7d9ef","Type":"ContainerDied","Data":"394baafb165a43b41a7166d6a509ff17de3d6315e81e688e58fca78148fb1f51"}
Oct 31 01:36:23.025076 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.025051    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"
Oct 31 01:36:23.031959 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.031924    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2" podUID="396b763c-807d-4f81-9bbc-20ba4696fee4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.102:50052\" within 1s: context deadline exceeded"
Oct 31 01:36:23.033404 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.033380    4476 scope.go:122] "RemoveContainer" containerID="281adc4625539d87afc32a363b73808ca9691320747a6db3700a6b3bb6e134b5"
Oct 31 01:36:23.033673 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:36:23.033654    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281adc4625539d87afc32a363b73808ca9691320747a6db3700a6b3bb6e134b5\": container with ID starting with 281adc4625539d87afc32a363b73808ca9691320747a6db3700a6b3bb6e134b5 not found: ID does not exist" containerID="281adc4625539d87afc32a363b73808ca9691320747a6db3700a6b3bb6e134b5"
Oct 31 01:36:23.033755 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.033685    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281adc4625539d87afc32a363b73808ca9691320747a6db3700a6b3bb6e134b5"} err="failed to get container status \"281adc4625539d87afc32a363b73808ca9691320747a6db3700a6b3bb6e134b5\": rpc error: code = NotFound desc = could not find container \"281adc4625539d87afc32a363b73808ca9691320747a6db3700a6b3bb6e134b5\": container with ID starting with 281adc4625539d87afc32a363b73808ca9691320747a6db3700a6b3bb6e134b5 not found: ID does not exist"
Oct 31 01:36:23.033755 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.033710    4476 scope.go:122] "RemoveContainer" containerID="55681d328992489406079f5eb4bad445b4d2d4b518a4f08bbaa8176a07256375"
Oct 31 01:36:23.185752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.185679    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"]
Oct 31 01:36:23.189903 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.189881    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c0359f-786c8479-t5j58"]
Oct 31 01:36:23.298079 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.298047    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"]
Oct 31 01:36:23.301481 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:23.301450    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c0359f-7f8cfd7687-82zx2"]
Oct 31 01:36:24.455072 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:24.455033    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396b763c-807d-4f81-9bbc-20ba4696fee4" path="/var/lib/kubelet/pods/396b763c-807d-4f81-9bbc-20ba4696fee4/volumes"
Oct 31 01:36:24.455597 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:24.455569    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddcfb701-535b-4321-94e2-d64efff7d9ef" path="/var/lib/kubelet/pods/ddcfb701-535b-4321-94e2-d64efff7d9ef/volumes"
Oct 31 01:36:25.651914 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:25.651871    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:29.181654 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:36:29.181617    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:36:29.561583 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:29.561553    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:30.036815 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:30.036777    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:30.736157 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:30.736129    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:36:30.766887 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:30.766862    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:36:32.684202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:32.681695    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:33.511163 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:33.511128    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:37.382859 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:37.382829    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:38.339582 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:38.339545    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:41.148890 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:41.148854    4476 generic.go:356] "Generic (PLEG): container finished" podID="a535f499-71aa-4da2-823a-16d687bd5d92" containerID="c4d60dffbc9a377a159ec1828d47e4fa900d14243a1c58a247506fd48361c480" exitCode=0
Oct 31 01:36:41.149365 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:41.148925    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9" event={"ID":"a535f499-71aa-4da2-823a-16d687bd5d92","Type":"ContainerDied","Data":"c4d60dffbc9a377a159ec1828d47e4fa900d14243a1c58a247506fd48361c480"}
Oct 31 01:36:41.149424 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:41.149379    4476 scope.go:122] "RemoveContainer" containerID="c4d60dffbc9a377a159ec1828d47e4fa900d14243a1c58a247506fd48361c480"
Oct 31 01:36:41.151142 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:41.151115    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:36:41.151248 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:41.151213    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:36:41.151289 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:41.151251    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:36:42.156851 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:42.156817    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9" event={"ID":"a535f499-71aa-4da2-823a-16d687bd5d92","Type":"ContainerStarted","Data":"27ed3c911be4746de61750ad721f406c771a6147ef6a513594965108168a8263"}
Oct 31 01:36:42.344252 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:42.344220    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:42.519141 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:42.519051    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:36:42.519343 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:42.519155    4476 patch_prober.go:29] interesting pod/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:36:42.519343 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:42.519245    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9" podUID="a535f499-71aa-4da2-823a-16d687bd5d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:36:43.519369 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:43.519332    4476 patch_prober.go:29] interesting pod/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:36:43.519770 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:43.519384    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9" podUID="a535f499-71aa-4da2-823a-16d687bd5d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:36:44.518854 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:44.518819    4476 patch_prober.go:29] interesting pod/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:36:44.519043 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:44.518880    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9" podUID="a535f499-71aa-4da2-823a-16d687bd5d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:36:45.392564 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:45.392526    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"]
Oct 31 01:36:45.503567 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:45.503532    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"]
Oct 31 01:36:45.518532 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:45.518499    4476 patch_prober.go:29] interesting pod/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:36:45.518715 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:45.518563    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9" podUID="a535f499-71aa-4da2-823a-16d687bd5d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:36:45.703609 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:45.703532    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:45.951715 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:45.951685    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:46.170953 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:46.170902    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:46.519334 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:46.519237    4476 patch_prober.go:29] interesting pod/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:36:46.519334 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:46.519297    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9" podUID="a535f499-71aa-4da2-823a-16d687bd5d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:36:47.036825 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:47.036757    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:47.523636 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:47.523563    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:36:47.524094 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:47.524077    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:36:47.524676 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:47.524655    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:36:48.943849 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:48.943812    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"]
Oct 31 01:36:48.944303 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:48.944020    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r" podUID="db07ee58-a36d-485b-aaf1-9f01749469e8" containerName="istio-proxy" containerID="cri-o://68f27dbd91878b9748d212ae74c2fa49d582135c3ec1721b96854cec1cb8ca45" gracePeriod=30
Oct 31 01:36:48.993411 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:48.993380    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"]
Oct 31 01:36:48.993636 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:48.993612    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd" podUID="8e2f4139-a07c-4159-adb4-8fbad0fe1897" containerName="istio-proxy" containerID="cri-o://89d9c61539e148ce5b133155ea421d08c1d4e529561db6ca71c42caf00ad1cab" gracePeriod=30
Oct 31 01:36:49.111590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:49.111549    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:49.809135 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:49.809105    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:36:49.809317 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:49.809150    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd" podUID="8e2f4139-a07c-4159-adb4-8fbad0fe1897" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:36:50.392819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.392777    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4" podUID="c2f25d63-5370-41c8-a8cd-5843ea7e11f6" containerName="payload-processing" containerID="cri-o://df78a93f56fb14e6d09f010f11958ccf2108a42647fb136fffdf260b283b4b4b" gracePeriod=25
Oct 31 01:36:50.504192 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.504148    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k" podUID="c664aa20-87b7-4ed6-8b7e-d86a5dca7f92" containerName="payload-pre-processing" containerID="cri-o://47c74f6f1ed596e89f2a2054dc41afdb4581e48974dcfe9afb3d07f282e2b4ec" gracePeriod=25
Oct 31 01:36:50.655250 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.655168    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"
Oct 31 01:36:50.699517 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.699486    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f25d63-5370-41c8-a8cd-5843ea7e11f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f25d63-5370-41c8-a8cd-5843ea7e11f6-plugins-config-volume\") pod \"c2f25d63-5370-41c8-a8cd-5843ea7e11f6\" (UID: \"c2f25d63-5370-41c8-a8cd-5843ea7e11f6\") "
Oct 31 01:36:50.699680 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.699539    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f25d63-5370-41c8-a8cd-5843ea7e11f6-kube-api-access-fblmv\" (UniqueName: \"kubernetes.io/projected/c2f25d63-5370-41c8-a8cd-5843ea7e11f6-kube-api-access-fblmv\") pod \"c2f25d63-5370-41c8-a8cd-5843ea7e11f6\" (UID: \"c2f25d63-5370-41c8-a8cd-5843ea7e11f6\") "
Oct 31 01:36:50.700121 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.700081    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f25d63-5370-41c8-a8cd-5843ea7e11f6-plugins-config-volume" pod "c2f25d63-5370-41c8-a8cd-5843ea7e11f6" (UID: "c2f25d63-5370-41c8-a8cd-5843ea7e11f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:50.702292 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.702267    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f25d63-5370-41c8-a8cd-5843ea7e11f6-kube-api-access-fblmv" pod "c2f25d63-5370-41c8-a8cd-5843ea7e11f6" (UID: "c2f25d63-5370-41c8-a8cd-5843ea7e11f6"). InnerVolumeSpecName "kube-api-access-fblmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:50.755199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.755153    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"
Oct 31 01:36:50.800323 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.800296    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92-kube-api-access-whvrd\" (UniqueName: \"kubernetes.io/projected/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92-kube-api-access-whvrd\") pod \"c664aa20-87b7-4ed6-8b7e-d86a5dca7f92\" (UID: \"c664aa20-87b7-4ed6-8b7e-d86a5dca7f92\") "
Oct 31 01:36:50.800511 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.800493    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92-plugins-config-volume\") pod \"c664aa20-87b7-4ed6-8b7e-d86a5dca7f92\" (UID: \"c664aa20-87b7-4ed6-8b7e-d86a5dca7f92\") "
Oct 31 01:36:50.800846 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.800827    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f25d63-5370-41c8-a8cd-5843ea7e11f6-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:50.800846 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.800845    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fblmv\" (UniqueName: \"kubernetes.io/projected/c2f25d63-5370-41c8-a8cd-5843ea7e11f6-kube-api-access-fblmv\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:50.801207 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.800889    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92-plugins-config-volume" pod "c664aa20-87b7-4ed6-8b7e-d86a5dca7f92" (UID: "c664aa20-87b7-4ed6-8b7e-d86a5dca7f92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:50.802516 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.802493    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92-kube-api-access-whvrd" pod "c664aa20-87b7-4ed6-8b7e-d86a5dca7f92" (UID: "c664aa20-87b7-4ed6-8b7e-d86a5dca7f92"). InnerVolumeSpecName "kube-api-access-whvrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:50.902325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.902296    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whvrd\" (UniqueName: \"kubernetes.io/projected/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92-kube-api-access-whvrd\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:50.902325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:50.902326    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:51.237485 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.237453    4476 generic.go:356] "Generic (PLEG): container finished" podID="c664aa20-87b7-4ed6-8b7e-d86a5dca7f92" containerID="47c74f6f1ed596e89f2a2054dc41afdb4581e48974dcfe9afb3d07f282e2b4ec" exitCode=0
Oct 31 01:36:51.237701 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.237557    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"
Oct 31 01:36:51.237701 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.237586    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k" event={"ID":"c664aa20-87b7-4ed6-8b7e-d86a5dca7f92","Type":"ContainerDied","Data":"47c74f6f1ed596e89f2a2054dc41afdb4581e48974dcfe9afb3d07f282e2b4ec"}
Oct 31 01:36:51.237701 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.237669    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k" event={"ID":"c664aa20-87b7-4ed6-8b7e-d86a5dca7f92","Type":"ContainerDied","Data":"164ec3792e855655b19c5ef2f8aab4bb723b10c19cc9e20eca3d99d835ff2609"}
Oct 31 01:36:51.237701 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.237700    4476 scope.go:122] "RemoveContainer" containerID="47c74f6f1ed596e89f2a2054dc41afdb4581e48974dcfe9afb3d07f282e2b4ec"
Oct 31 01:36:51.239219 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.238919    4476 generic.go:356] "Generic (PLEG): container finished" podID="c2f25d63-5370-41c8-a8cd-5843ea7e11f6" containerID="df78a93f56fb14e6d09f010f11958ccf2108a42647fb136fffdf260b283b4b4b" exitCode=0
Oct 31 01:36:51.239219 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.238958    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"
Oct 31 01:36:51.239219 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.238973    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4" event={"ID":"c2f25d63-5370-41c8-a8cd-5843ea7e11f6","Type":"ContainerDied","Data":"df78a93f56fb14e6d09f010f11958ccf2108a42647fb136fffdf260b283b4b4b"}
Oct 31 01:36:51.239219 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.239019    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4" event={"ID":"c2f25d63-5370-41c8-a8cd-5843ea7e11f6","Type":"ContainerDied","Data":"cd98f320788ec598a86f84c803123678b2376033c4702f4f5262944aa85a1d5d"}
Oct 31 01:36:51.251442 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.251418    4476 scope.go:122] "RemoveContainer" containerID="47c74f6f1ed596e89f2a2054dc41afdb4581e48974dcfe9afb3d07f282e2b4ec"
Oct 31 01:36:51.251703 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:36:51.251685    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c74f6f1ed596e89f2a2054dc41afdb4581e48974dcfe9afb3d07f282e2b4ec\": container with ID starting with 47c74f6f1ed596e89f2a2054dc41afdb4581e48974dcfe9afb3d07f282e2b4ec not found: ID does not exist" containerID="47c74f6f1ed596e89f2a2054dc41afdb4581e48974dcfe9afb3d07f282e2b4ec"
Oct 31 01:36:51.251771 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.251709    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c74f6f1ed596e89f2a2054dc41afdb4581e48974dcfe9afb3d07f282e2b4ec"} err="failed to get container status \"47c74f6f1ed596e89f2a2054dc41afdb4581e48974dcfe9afb3d07f282e2b4ec\": rpc error: code = NotFound desc = could not find container \"47c74f6f1ed596e89f2a2054dc41afdb4581e48974dcfe9afb3d07f282e2b4ec\": container with ID starting with 47c74f6f1ed596e89f2a2054dc41afdb4581e48974dcfe9afb3d07f282e2b4ec not found: ID does not exist"
Oct 31 01:36:51.251771 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.251725    4476 scope.go:122] "RemoveContainer" containerID="df78a93f56fb14e6d09f010f11958ccf2108a42647fb136fffdf260b283b4b4b"
Oct 31 01:36:51.260862 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.260847    4476 scope.go:122] "RemoveContainer" containerID="df78a93f56fb14e6d09f010f11958ccf2108a42647fb136fffdf260b283b4b4b"
Oct 31 01:36:51.261120 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:36:51.261104    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df78a93f56fb14e6d09f010f11958ccf2108a42647fb136fffdf260b283b4b4b\": container with ID starting with df78a93f56fb14e6d09f010f11958ccf2108a42647fb136fffdf260b283b4b4b not found: ID does not exist" containerID="df78a93f56fb14e6d09f010f11958ccf2108a42647fb136fffdf260b283b4b4b"
Oct 31 01:36:51.261202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.261130    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df78a93f56fb14e6d09f010f11958ccf2108a42647fb136fffdf260b283b4b4b"} err="failed to get container status \"df78a93f56fb14e6d09f010f11958ccf2108a42647fb136fffdf260b283b4b4b\": rpc error: code = NotFound desc = could not find container \"df78a93f56fb14e6d09f010f11958ccf2108a42647fb136fffdf260b283b4b4b\": container with ID starting with df78a93f56fb14e6d09f010f11958ccf2108a42647fb136fffdf260b283b4b4b not found: ID does not exist"
Oct 31 01:36:51.399096 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.399071    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"]
Oct 31 01:36:51.399642 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.399627    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c664aa20-87b7-4ed6-8b7e-d86a5dca7f92" containerName="payload-pre-processing"
Oct 31 01:36:51.399692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.399644    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c664aa20-87b7-4ed6-8b7e-d86a5dca7f92" containerName="payload-pre-processing"
Oct 31 01:36:51.399692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.399658    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ddcfb701-535b-4321-94e2-d64efff7d9ef" containerName="payload-processing"
Oct 31 01:36:51.399692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.399663    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddcfb701-535b-4321-94e2-d64efff7d9ef" containerName="payload-processing"
Oct 31 01:36:51.399692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.399675    4476 cpu_manager.go:414] "containerMap: removing container" podUID="396b763c-807d-4f81-9bbc-20ba4696fee4" containerName="payload-pre-processing"
Oct 31 01:36:51.399692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.399680    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396b763c-807d-4f81-9bbc-20ba4696fee4" containerName="payload-pre-processing"
Oct 31 01:36:51.399842 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.399703    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c2f25d63-5370-41c8-a8cd-5843ea7e11f6" containerName="payload-processing"
Oct 31 01:36:51.399842 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.399708    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f25d63-5370-41c8-a8cd-5843ea7e11f6" containerName="payload-processing"
Oct 31 01:36:51.399842 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.399779    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="396b763c-807d-4f81-9bbc-20ba4696fee4" containerName="payload-pre-processing"
Oct 31 01:36:51.399842 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.399789    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f25d63-5370-41c8-a8cd-5843ea7e11f6" containerName="payload-processing"
Oct 31 01:36:51.399842 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.399798    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c664aa20-87b7-4ed6-8b7e-d86a5dca7f92" containerName="payload-pre-processing"
Oct 31 01:36:51.399842 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.399803    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddcfb701-535b-4321-94e2-d64efff7d9ef" containerName="payload-processing"
Oct 31 01:36:51.404204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.404186    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:51.406061 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.406041    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"]
Oct 31 01:36:51.448641 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.448620    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-634135-6c54f6fdbc-nkpz4"]
Oct 31 01:36:51.457981 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.457960    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3f0818-serving-cert\""
Oct 31 01:36:51.461530 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.461492    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"]
Oct 31 01:36:51.508820 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.508735    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-metrics-certs\") pod \"maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:51.509163 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.509138    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5cc350-c137-491c-94e2-efafdfb82a0f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:51.509294 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.509169    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6lln\" (UniqueName: \"kubernetes.io/projected/bf5cc350-c137-491c-94e2-efafdfb82a0f-kube-api-access-d6lln\") pod \"maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:51.509363 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.509335    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:51.570144 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.570111    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"]
Oct 31 01:36:51.577838 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.577810    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-634135-8cd476b87-s6c5k"]
Oct 31 01:36:51.611041 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.611013    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-metrics-certs\") pod \"maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:51.611190 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.611099    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5cc350-c137-491c-94e2-efafdfb82a0f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:51.611190 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.611115    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6lln\" (UniqueName: \"kubernetes.io/projected/bf5cc350-c137-491c-94e2-efafdfb82a0f-kube-api-access-d6lln\") pod \"maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:51.611190 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.611147    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:51.611382 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:36:51.611343    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-serving-cert: secret "maas-api-e2e-ipp-b-3f0818-serving-cert" not found
Oct 31 01:36:51.611443 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:36:51.611421    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-maas-api-tls podName:bf5cc350-c137-491c-94e2-efafdfb82a0f nodeName:}" failed. No retries permitted until 2026-10-31 01:36:52.111392721 +0000 UTC m=+2421.801275836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-maas-api-tls") pod "maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4" (UID: "bf5cc350-c137-491c-94e2-efafdfb82a0f") : secret "maas-api-e2e-ipp-b-3f0818-serving-cert" not found
Oct 31 01:36:51.611813 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.611790    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5cc350-c137-491c-94e2-efafdfb82a0f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:51.613600 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.613581    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-metrics-certs\") pod \"maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:51.680376 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.676926    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6lln\" (UniqueName: \"kubernetes.io/projected/bf5cc350-c137-491c-94e2-efafdfb82a0f-kube-api-access-d6lln\") pod \"maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:51.695716 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.695686    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:51.879939 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:51.879890    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:52.055000 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:52.054952    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:52.117814 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:52.117779    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:52.120352 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:52.120323    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:52.326938 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:52.326910    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:52.454883 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:52.454855    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f25d63-5370-41c8-a8cd-5843ea7e11f6" path="/var/lib/kubelet/pods/c2f25d63-5370-41c8-a8cd-5843ea7e11f6/volumes"
Oct 31 01:36:52.455316 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:52.455254    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c664aa20-87b7-4ed6-8b7e-d86a5dca7f92" path="/var/lib/kubelet/pods/c664aa20-87b7-4ed6-8b7e-d86a5dca7f92/volumes"
Oct 31 01:36:52.713972 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:52.713945    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"]
Oct 31 01:36:52.714583 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:36:52.714561    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5cc350_c137_491c_94e2_efafdfb82a0f.slice/crio-1a99f77b3deb1847bb7524670b4426d7df90204422d2c2f977fe5164d47f3e0d WatchSource:0}: Error finding container 1a99f77b3deb1847bb7524670b4426d7df90204422d2c2f977fe5164d47f3e0d: Status 404 returned error can't find the container with id 1a99f77b3deb1847bb7524670b4426d7df90204422d2c2f977fe5164d47f3e0d
Oct 31 01:36:53.253739 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:53.253701    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4" event={"ID":"bf5cc350-c137-491c-94e2-efafdfb82a0f","Type":"ContainerStarted","Data":"1a99f77b3deb1847bb7524670b4426d7df90204422d2c2f977fe5164d47f3e0d"}
Oct 31 01:36:53.957367 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:53.957273    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": read tcp 10.133.0.2:41504->10.133.0.82:15021: read: connection reset by peer" start-of-body=
Oct 31 01:36:53.957367 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:53.957340    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r" podUID="db07ee58-a36d-485b-aaf1-9f01749469e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": read tcp 10.133.0.2:41504->10.133.0.82:15021: read: connection reset by peer"
Oct 31 01:36:54.028473 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.028448    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:54.263021 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.262985    4476 generic.go:356] "Generic (PLEG): container finished" podID="8e2f4139-a07c-4159-adb4-8fbad0fe1897" containerID="89d9c61539e148ce5b133155ea421d08c1d4e529561db6ca71c42caf00ad1cab" exitCode=0
Oct 31 01:36:54.263227 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.263058    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd" event={"ID":"8e2f4139-a07c-4159-adb4-8fbad0fe1897","Type":"ContainerDied","Data":"89d9c61539e148ce5b133155ea421d08c1d4e529561db6ca71c42caf00ad1cab"}
Oct 31 01:36:54.264889 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.264860    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4" event={"ID":"bf5cc350-c137-491c-94e2-efafdfb82a0f","Type":"ContainerStarted","Data":"fc72b336c7c781567bde23bbdf156f83dfc550a1df7fafa17ce4a0fc50882a3c"}
Oct 31 01:36:54.265030 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.265003    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:36:54.266359 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.266326    4476 generic.go:356] "Generic (PLEG): container finished" podID="db07ee58-a36d-485b-aaf1-9f01749469e8" containerID="68f27dbd91878b9748d212ae74c2fa49d582135c3ec1721b96854cec1cb8ca45" exitCode=0
Oct 31 01:36:54.266455 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.266380    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r" event={"ID":"db07ee58-a36d-485b-aaf1-9f01749469e8","Type":"ContainerDied","Data":"68f27dbd91878b9748d212ae74c2fa49d582135c3ec1721b96854cec1cb8ca45"}
Oct 31 01:36:54.266455 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.266410    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r" event={"ID":"db07ee58-a36d-485b-aaf1-9f01749469e8","Type":"ContainerDied","Data":"007979ee3dcf8d26fb36dd96dc56c4ae07b229b44aa51d6692c15d43cf4835c6"}
Oct 31 01:36:54.266455 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.266427    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="007979ee3dcf8d26fb36dd96dc56c4ae07b229b44aa51d6692c15d43cf4835c6"
Oct 31 01:36:54.267587 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.267570    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:36:54.283258 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.283233    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:36:54.346573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.346540    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-credential-socket\") pod \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") "
Oct 31 01:36:54.346573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.346575    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-workload-socket\") pod \"db07ee58-a36d-485b-aaf1-9f01749469e8\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") "
Oct 31 01:36:54.346803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.346653    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-podinfo\") pod \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") "
Oct 31 01:36:54.346803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.346680    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-envoy\") pod \"db07ee58-a36d-485b-aaf1-9f01749469e8\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") "
Oct 31 01:36:54.346803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.346717    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2f4139-a07c-4159-adb4-8fbad0fe1897-kube-api-access-x9rsh\" (UniqueName: \"kubernetes.io/projected/8e2f4139-a07c-4159-adb4-8fbad0fe1897-kube-api-access-x9rsh\") pod \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") "
Oct 31 01:36:54.346803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.346753    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-data\") pod \"db07ee58-a36d-485b-aaf1-9f01749469e8\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") "
Oct 31 01:36:54.347027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.346801    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-workload-certs\") pod \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") "
Oct 31 01:36:54.347027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.346843    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db07ee58-a36d-485b-aaf1-9f01749469e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db07ee58-a36d-485b-aaf1-9f01749469e8-istiod-ca-cert\") pod \"db07ee58-a36d-485b-aaf1-9f01749469e8\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") "
Oct 31 01:36:54.347027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.346849    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-credential-socket" pod "8e2f4139-a07c-4159-adb4-8fbad0fe1897" (UID: "8e2f4139-a07c-4159-adb4-8fbad0fe1897"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:54.347027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.346898    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-credential-socket\") pod \"db07ee58-a36d-485b-aaf1-9f01749469e8\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") "
Oct 31 01:36:54.347027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.346924    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-podinfo\") pod \"db07ee58-a36d-485b-aaf1-9f01749469e8\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") "
Oct 31 01:36:54.347027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.346928    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-workload-socket" pod "db07ee58-a36d-485b-aaf1-9f01749469e8" (UID: "db07ee58-a36d-485b-aaf1-9f01749469e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:54.347027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.346972    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-workload-certs\") pod \"db07ee58-a36d-485b-aaf1-9f01749469e8\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") "
Oct 31 01:36:54.347027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.347012    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istiod-ca-cert\") pod \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") "
Oct 31 01:36:54.347643 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.347080    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-token\" (UniqueName: \"kubernetes.io/projected/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-token\") pod \"db07ee58-a36d-485b-aaf1-9f01749469e8\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") "
Oct 31 01:36:54.347643 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.347084    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-workload-certs" pod "8e2f4139-a07c-4159-adb4-8fbad0fe1897" (UID: "8e2f4139-a07c-4159-adb4-8fbad0fe1897"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:54.347643 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.347093    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-data" pod "db07ee58-a36d-485b-aaf1-9f01749469e8" (UID: "db07ee58-a36d-485b-aaf1-9f01749469e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:54.347643 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.347125    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-workload-socket\") pod \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") "
Oct 31 01:36:54.347643 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.347235    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db07ee58-a36d-485b-aaf1-9f01749469e8-kube-api-access-fz68w\" (UniqueName: \"kubernetes.io/projected/db07ee58-a36d-485b-aaf1-9f01749469e8-kube-api-access-fz68w\") pod \"db07ee58-a36d-485b-aaf1-9f01749469e8\" (UID: \"db07ee58-a36d-485b-aaf1-9f01749469e8\") "
Oct 31 01:36:54.347643 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.347271    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-token\" (UniqueName: \"kubernetes.io/projected/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-token\") pod \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") "
Oct 31 01:36:54.347643 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.347301    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-data\") pod \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") "
Oct 31 01:36:54.347643 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.347334    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-workload-socket" pod "8e2f4139-a07c-4159-adb4-8fbad0fe1897" (UID: "8e2f4139-a07c-4159-adb4-8fbad0fe1897"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:54.347643 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.347343    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-envoy\") pod \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\" (UID: \"8e2f4139-a07c-4159-adb4-8fbad0fe1897\") "
Oct 31 01:36:54.347643 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.347357    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db07ee58-a36d-485b-aaf1-9f01749469e8-istiod-ca-cert" pod "db07ee58-a36d-485b-aaf1-9f01749469e8" (UID: "db07ee58-a36d-485b-aaf1-9f01749469e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:54.347643 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.347502    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-workload-certs" pod "db07ee58-a36d-485b-aaf1-9f01749469e8" (UID: "db07ee58-a36d-485b-aaf1-9f01749469e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:54.347643 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.347528    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-credential-socket" pod "db07ee58-a36d-485b-aaf1-9f01749469e8" (UID: "db07ee58-a36d-485b-aaf1-9f01749469e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:54.348296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.347775    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istiod-ca-cert" pod "8e2f4139-a07c-4159-adb4-8fbad0fe1897" (UID: "8e2f4139-a07c-4159-adb4-8fbad0fe1897"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:54.348296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.348155    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.348296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.348199    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.348296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.348216    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.348296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.348217    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-data" pod "8e2f4139-a07c-4159-adb4-8fbad0fe1897" (UID: "8e2f4139-a07c-4159-adb4-8fbad0fe1897"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:54.348296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.348232    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.348296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.348247    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.348296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.348261    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.348296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.348275    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.348296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.348288    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db07ee58-a36d-485b-aaf1-9f01749469e8-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.348296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.348302    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.349849 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.349819    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2f4139-a07c-4159-adb4-8fbad0fe1897-kube-api-access-x9rsh" pod "8e2f4139-a07c-4159-adb4-8fbad0fe1897" (UID: "8e2f4139-a07c-4159-adb4-8fbad0fe1897"). InnerVolumeSpecName "kube-api-access-x9rsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:54.350248 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.350222    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-token" pod "8e2f4139-a07c-4159-adb4-8fbad0fe1897" (UID: "8e2f4139-a07c-4159-adb4-8fbad0fe1897"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:54.350426 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.350406    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-podinfo" pod "db07ee58-a36d-485b-aaf1-9f01749469e8" (UID: "db07ee58-a36d-485b-aaf1-9f01749469e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:36:54.350708 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.350688    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-envoy" pod "db07ee58-a36d-485b-aaf1-9f01749469e8" (UID: "db07ee58-a36d-485b-aaf1-9f01749469e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:54.350769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.350737    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-podinfo" pod "8e2f4139-a07c-4159-adb4-8fbad0fe1897" (UID: "8e2f4139-a07c-4159-adb4-8fbad0fe1897"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:36:54.350838 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.350778    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-token" pod "db07ee58-a36d-485b-aaf1-9f01749469e8" (UID: "db07ee58-a36d-485b-aaf1-9f01749469e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:54.350884 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.350828    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-envoy" pod "8e2f4139-a07c-4159-adb4-8fbad0fe1897" (UID: "8e2f4139-a07c-4159-adb4-8fbad0fe1897"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:54.351014 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.350997    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db07ee58-a36d-485b-aaf1-9f01749469e8-kube-api-access-fz68w" pod "db07ee58-a36d-485b-aaf1-9f01749469e8" (UID: "db07ee58-a36d-485b-aaf1-9f01749469e8"). InnerVolumeSpecName "kube-api-access-fz68w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:54.380870 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.380826    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4" podStartSLOduration=2.555250768 podStartE2EDuration="3.380812728s" totalImagesPullingTime="825.56196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:54.380100941 +0000 UTC m=+2424.069984082" watchObservedRunningTime="2026-10-31 01:36:54.380812728 +0000 UTC m=+2424.070695854"
Oct 31 01:36:54.449084 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.449052    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.449084 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.449080    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.449084 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.449090    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9rsh\" (UniqueName: \"kubernetes.io/projected/8e2f4139-a07c-4159-adb4-8fbad0fe1897-kube-api-access-x9rsh\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.449323 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.449099    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.449323 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.449108    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db07ee58-a36d-485b-aaf1-9f01749469e8-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.449323 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.449117    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz68w\" (UniqueName: \"kubernetes.io/projected/db07ee58-a36d-485b-aaf1-9f01749469e8-kube-api-access-fz68w\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.449323 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.449124    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.449323 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.449133    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.449323 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.449142    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2f4139-a07c-4159-adb4-8fbad0fe1897-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:54.859353 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:54.859324    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:55.273074 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:55.272988    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"
Oct 31 01:36:55.273074 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:55.273003    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd" event={"ID":"8e2f4139-a07c-4159-adb4-8fbad0fe1897","Type":"ContainerDied","Data":"80b1124627265fb974ccf131f2c1ea9672ef5c6d64fe2ff80c3bb9767d2166f6"}
Oct 31 01:36:55.273653 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:55.273100    4476 scope.go:122] "RemoveContainer" containerID="89d9c61539e148ce5b133155ea421d08c1d4e529561db6ca71c42caf00ad1cab"
Oct 31 01:36:55.273653 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:55.273202    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"
Oct 31 01:36:55.437696 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:55.437660    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"]
Oct 31 01:36:55.487020 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:55.486978    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-nvq4r"]
Oct 31 01:36:55.549703 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:55.549670    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"]
Oct 31 01:36:55.553081 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:55.553053    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-7g4sd"]
Oct 31 01:36:56.042153 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:56.042124    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:56.458350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:56.458272    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2f4139-a07c-4159-adb4-8fbad0fe1897" path="/var/lib/kubelet/pods/8e2f4139-a07c-4159-adb4-8fbad0fe1897/volumes"
Oct 31 01:36:56.458759 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:56.458745    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db07ee58-a36d-485b-aaf1-9f01749469e8" path="/var/lib/kubelet/pods/db07ee58-a36d-485b-aaf1-9f01749469e8/volumes"
Oct 31 01:36:56.553555 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:56.553515    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:56.731976 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:56.731891    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:57.566577 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:57.566547    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:57.850516 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:57.850310    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:58.037535 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:58.037496    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:58.634925 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:36:58.634891    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:59.181409 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:36:59.181372    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:37:00.278971 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:00.278939    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:37:01.766108 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:01.766076    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:02.275643 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:02.274992    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:04.598997 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:04.598962    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:04.820099 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:04.820062    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:05.026375 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:05.026274    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:05.706404 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:05.706367    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:06.417486 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:06.417452    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:10.084923 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:10.084889    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:13.942268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:13.942227    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:17.891137 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:17.891104    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:23.337137 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:23.337101    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:28.093907 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:28.093877    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:29.181873 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:37:29.181837    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:37:30.995342 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:30.995311    4476 scope.go:122] "RemoveContainer" containerID="68f27dbd91878b9748d212ae74c2fa49d582135c3ec1721b96854cec1cb8ca45"
Oct 31 01:37:32.389402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:32.389371    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:33.849647 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:33.849614    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:34.027771 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:34.027735    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:34.217332 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:34.217258    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:34.400117 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:34.400080    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:34.969395 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:34.969367    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:38.669134 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:38.669090    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:38.855290 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:38.855260    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:39.541949 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:39.541911    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:40.790975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:40.790894    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:40.970771 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:40.970712    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:41.181626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:41.181537    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:41.658664 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:41.658612    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:43.324732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:43.324703    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"]
Oct 31 01:37:43.622119 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:43.621999    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:43.812288 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:43.812257    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:45.505999 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:45.505971    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vl9ww_c0881ec8-1f45-459a-b40c-29e2d2246ea0/payload-processing/0.log"
Oct 31 01:37:45.741391 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:45.741363    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vl9ww_c0881ec8-1f45-459a-b40c-29e2d2246ea0/payload-processing/0.log"
Oct 31 01:37:46.485979 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:46.485925    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:48.325723 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.325674    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq" podUID="1140ac0a-116a-4f86-9f5d-803e1f3a4667" containerName="payload-pre-processing" containerID="cri-o://727c36125b8f64d53b1931de4ea05d65246c3e33466641c6b8d819a1406d5d8d" gracePeriod=25
Oct 31 01:37:48.496613 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:37:48.496584    4476 logging.go:55] [core] [Channel #3041 SubChannel #3042]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.108:50052", ServerName: "10.133.0.108:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.108:50052: connect: connection refused"
Oct 31 01:37:48.586747 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.586693    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"
Oct 31 01:37:48.594390 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.594363    4476 generic.go:356] "Generic (PLEG): container finished" podID="1140ac0a-116a-4f86-9f5d-803e1f3a4667" containerID="727c36125b8f64d53b1931de4ea05d65246c3e33466641c6b8d819a1406d5d8d" exitCode=0
Oct 31 01:37:48.594509 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.594395    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq" event={"ID":"1140ac0a-116a-4f86-9f5d-803e1f3a4667","Type":"ContainerDied","Data":"727c36125b8f64d53b1931de4ea05d65246c3e33466641c6b8d819a1406d5d8d"}
Oct 31 01:37:48.594509 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.594446    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq" event={"ID":"1140ac0a-116a-4f86-9f5d-803e1f3a4667","Type":"ContainerDied","Data":"222e4411b0d009f06bdea3fcc4c7f34b2b7ecbf23872a12d844b4ec097c6d74a"}
Oct 31 01:37:48.594509 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.594467    4476 scope.go:122] "RemoveContainer" containerID="727c36125b8f64d53b1931de4ea05d65246c3e33466641c6b8d819a1406d5d8d"
Oct 31 01:37:48.594509 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.594401    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"
Oct 31 01:37:48.603255 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.603240    4476 scope.go:122] "RemoveContainer" containerID="727c36125b8f64d53b1931de4ea05d65246c3e33466641c6b8d819a1406d5d8d"
Oct 31 01:37:48.603497 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:37:48.603482    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727c36125b8f64d53b1931de4ea05d65246c3e33466641c6b8d819a1406d5d8d\": container with ID starting with 727c36125b8f64d53b1931de4ea05d65246c3e33466641c6b8d819a1406d5d8d not found: ID does not exist" containerID="727c36125b8f64d53b1931de4ea05d65246c3e33466641c6b8d819a1406d5d8d"
Oct 31 01:37:48.603554 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.603505    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727c36125b8f64d53b1931de4ea05d65246c3e33466641c6b8d819a1406d5d8d"} err="failed to get container status \"727c36125b8f64d53b1931de4ea05d65246c3e33466641c6b8d819a1406d5d8d\": rpc error: code = NotFound desc = could not find container \"727c36125b8f64d53b1931de4ea05d65246c3e33466641c6b8d819a1406d5d8d\": container with ID starting with 727c36125b8f64d53b1931de4ea05d65246c3e33466641c6b8d819a1406d5d8d not found: ID does not exist"
Oct 31 01:37:48.640802 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.640782    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1140ac0a-116a-4f86-9f5d-803e1f3a4667-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1140ac0a-116a-4f86-9f5d-803e1f3a4667-plugins-config-volume\") pod \"1140ac0a-116a-4f86-9f5d-803e1f3a4667\" (UID: \"1140ac0a-116a-4f86-9f5d-803e1f3a4667\") "
Oct 31 01:37:48.640929 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.640810    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1140ac0a-116a-4f86-9f5d-803e1f3a4667-kube-api-access-zb6dn\" (UniqueName: \"kubernetes.io/projected/1140ac0a-116a-4f86-9f5d-803e1f3a4667-kube-api-access-zb6dn\") pod \"1140ac0a-116a-4f86-9f5d-803e1f3a4667\" (UID: \"1140ac0a-116a-4f86-9f5d-803e1f3a4667\") "
Oct 31 01:37:48.641106 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.641087    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1140ac0a-116a-4f86-9f5d-803e1f3a4667-plugins-config-volume" pod "1140ac0a-116a-4f86-9f5d-803e1f3a4667" (UID: "1140ac0a-116a-4f86-9f5d-803e1f3a4667"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:37:48.641305 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.641278    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1140ac0a-116a-4f86-9f5d-803e1f3a4667-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:48.642916 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.642898    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1140ac0a-116a-4f86-9f5d-803e1f3a4667-kube-api-access-zb6dn" pod "1140ac0a-116a-4f86-9f5d-803e1f3a4667" (UID: "1140ac0a-116a-4f86-9f5d-803e1f3a4667"). InnerVolumeSpecName "kube-api-access-zb6dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:37:48.742375 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:48.742345    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb6dn\" (UniqueName: \"kubernetes.io/projected/1140ac0a-116a-4f86-9f5d-803e1f3a4667-kube-api-access-zb6dn\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:49.014624 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.014592    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"]
Oct 31 01:37:49.019682 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.019648    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq"]
Oct 31 01:37:49.450480 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.450450    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:49.497190 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.497141    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a31828-5d7786d855-ctlqq" podUID="1140ac0a-116a-4f86-9f5d-803e1f3a4667" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.108:50052\" within 1s: context deadline exceeded"
Oct 31 01:37:49.938262 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.938227    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"]
Oct 31 01:37:49.938713 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.938701    4476 cpu_manager.go:414] "containerMap: removing container" podUID="1140ac0a-116a-4f86-9f5d-803e1f3a4667" containerName="payload-pre-processing"
Oct 31 01:37:49.938757 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.938714    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1140ac0a-116a-4f86-9f5d-803e1f3a4667" containerName="payload-pre-processing"
Oct 31 01:37:49.938757 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.938729    4476 cpu_manager.go:414] "containerMap: removing container" podUID="db07ee58-a36d-485b-aaf1-9f01749469e8" containerName="istio-proxy"
Oct 31 01:37:49.938757 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.938734    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db07ee58-a36d-485b-aaf1-9f01749469e8" containerName="istio-proxy"
Oct 31 01:37:49.938757 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.938749    4476 cpu_manager.go:414] "containerMap: removing container" podUID="8e2f4139-a07c-4159-adb4-8fbad0fe1897" containerName="istio-proxy"
Oct 31 01:37:49.938757 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.938754    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2f4139-a07c-4159-adb4-8fbad0fe1897" containerName="istio-proxy"
Oct 31 01:37:49.938933 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.938834    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1140ac0a-116a-4f86-9f5d-803e1f3a4667" containerName="payload-pre-processing"
Oct 31 01:37:49.938933 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.938842    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="db07ee58-a36d-485b-aaf1-9f01749469e8" containerName="istio-proxy"
Oct 31 01:37:49.938933 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.938849    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2f4139-a07c-4159-adb4-8fbad0fe1897" containerName="istio-proxy"
Oct 31 01:37:49.943310 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:49.943295    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.045480 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.045450    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c9da8f\"/\"default-dockercfg-n4npn\""
Oct 31 01:37:50.045480 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.045466    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c9da8f\"/\"openshift-service-ca.crt\""
Oct 31 01:37:50.045480 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.045473    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c9da8f\"/\"kube-root-ca.crt\""
Oct 31 01:37:50.045801 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.045473    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c9da8f\"/\"ipp-route-c9da8f-kserve-self-signed-certs\""
Oct 31 01:37:50.048198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.048164    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"]
Oct 31 01:37:50.055185 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.055153    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvwvl\" (UniqueName: \"kubernetes.io/projected/6d153817-c799-4114-8018-4ee979d4bb03-kube-api-access-cvwvl\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.055279 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.055264    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-dshm\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.055371 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.055354    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-model-cache\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.055432 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.055413    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-tmp-dir\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.055548 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.055530    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-home\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.055581 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.055568    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d153817-c799-4114-8018-4ee979d4bb03-tls-certs\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.065311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.065291    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:50.156427 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.156395    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-dshm\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.156590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.156441    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-model-cache\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.156590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.156486    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-tmp-dir\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.156590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.156565    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-home\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.156752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.156600    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d153817-c799-4114-8018-4ee979d4bb03-tls-certs\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.156752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.156720    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvwvl\" (UniqueName: \"kubernetes.io/projected/6d153817-c799-4114-8018-4ee979d4bb03-kube-api-access-cvwvl\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.156965 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.156942    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-model-cache\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.157044 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.156976    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-tmp-dir\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.157215 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.157198    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-home\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.158829 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.158807    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-dshm\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.159351 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.159332    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d153817-c799-4114-8018-4ee979d4bb03-tls-certs\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.213317 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.213256    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvwvl\" (UniqueName: \"kubernetes.io/projected/6d153817-c799-4114-8018-4ee979d4bb03-kube-api-access-cvwvl\") pod \"ipp-route-c9da8f-kserve-7b587cbc89-9x8hj\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") " pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.254898 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.254875    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:50.453963 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.453928    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1140ac0a-116a-4f86-9f5d-803e1f3a4667" path="/var/lib/kubelet/pods/1140ac0a-116a-4f86-9f5d-803e1f3a4667/volumes"
Oct 31 01:37:50.631814 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:50.628779    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"]
Oct 31 01:37:51.620862 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:51.620818    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj" event={"ID":"6d153817-c799-4114-8018-4ee979d4bb03","Type":"ContainerStarted","Data":"cc32441e943b9a463c32527fdc41419922b5306e7ee7d6fb36eaffb7191557b9"}
Oct 31 01:37:51.621285 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:51.620871    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj" event={"ID":"6d153817-c799-4114-8018-4ee979d4bb03","Type":"ContainerStarted","Data":"4f736e8365e1f8ebcdd2f1159d3591f51a78d73477d3541d162ca2c43f089a26"}
Oct 31 01:37:51.735707 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:51.735658    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj" podStartSLOduration=2.735644815 podStartE2EDuration="2.735644815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:51.733831023 +0000 UTC m=+2481.423714152" watchObservedRunningTime="2026-10-31 01:37:51.735644815 +0000 UTC m=+2481.425527943"
Oct 31 01:37:52.255271 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:52.255240    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:37:52.657146 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:52.657117    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:53.044479 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:53.044440    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:53.285489 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:53.285449    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:53.465775 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:53.465658    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:56.601290 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:56.601258    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:57.369979 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:57.369950    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:58.448539 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:58.448503    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:58.654946 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:58.654915    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:58.846912 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:58.846882    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:59.181812 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:37:59.181735    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:37:59.959728 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:37:59.959673    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:00.256065 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:00.255987    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:38:00.268581 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:00.268556    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:38:00.689996 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:00.689970    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:38:01.680279 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:01.680246    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:01.860653 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:01.860626    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:02.046280 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:02.046242    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"]
Oct 31 01:38:02.050046 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:02.050030    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"
Oct 31 01:38:02.106445 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:02.106420    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"]
Oct 31 01:38:02.152369 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:02.152343    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-6a9710\""
Oct 31 01:38:02.152744 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:02.152706    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-6a9710-dockercfg-sf8lr\""
Oct 31 01:38:02.184376 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:02.184340    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98572193-ebb8-47e8-adc7-fbff21bc4c84-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8\" (UID: \"98572193-ebb8-47e8-adc7-fbff21bc4c84\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"
Oct 31 01:38:02.184536 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:02.184406    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2kgt\" (UniqueName: \"kubernetes.io/projected/98572193-ebb8-47e8-adc7-fbff21bc4c84-kube-api-access-x2kgt\") pod \"payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8\" (UID: \"98572193-ebb8-47e8-adc7-fbff21bc4c84\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"
Oct 31 01:38:02.285355 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:02.285326    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98572193-ebb8-47e8-adc7-fbff21bc4c84-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8\" (UID: \"98572193-ebb8-47e8-adc7-fbff21bc4c84\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"
Oct 31 01:38:02.285547 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:02.285396    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2kgt\" (UniqueName: \"kubernetes.io/projected/98572193-ebb8-47e8-adc7-fbff21bc4c84-kube-api-access-x2kgt\") pod \"payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8\" (UID: \"98572193-ebb8-47e8-adc7-fbff21bc4c84\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"
Oct 31 01:38:02.286035 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:02.286015    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98572193-ebb8-47e8-adc7-fbff21bc4c84-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8\" (UID: \"98572193-ebb8-47e8-adc7-fbff21bc4c84\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"
Oct 31 01:38:02.342977 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:02.342917    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2kgt\" (UniqueName: \"kubernetes.io/projected/98572193-ebb8-47e8-adc7-fbff21bc4c84-kube-api-access-x2kgt\") pod \"payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8\" (UID: \"98572193-ebb8-47e8-adc7-fbff21bc4c84\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"
Oct 31 01:38:02.363803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:02.363775    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"
Oct 31 01:38:02.735432 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:02.735345    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"]
Oct 31 01:38:02.740435 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:38:02.740408    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98572193_ebb8_47e8_adc7_fbff21bc4c84.slice/crio-d5837446cd7c99aba9b58597175722e823b5cb698c3736a72a4599e1774a2a4e WatchSource:0}: Error finding container d5837446cd7c99aba9b58597175722e823b5cb698c3736a72a4599e1774a2a4e: Status 404 returned error can't find the container with id d5837446cd7c99aba9b58597175722e823b5cb698c3736a72a4599e1774a2a4e
Oct 31 01:38:03.694105 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:03.694070    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8" event={"ID":"98572193-ebb8-47e8-adc7-fbff21bc4c84","Type":"ContainerStarted","Data":"7dfb1ee079cdd80969d8a3f8944fadbf378bbf30ae25d596f7c1413808a1bd96"}
Oct 31 01:38:03.694315 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:03.694127    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8" event={"ID":"98572193-ebb8-47e8-adc7-fbff21bc4c84","Type":"ContainerStarted","Data":"d5837446cd7c99aba9b58597175722e823b5cb698c3736a72a4599e1774a2a4e"}
Oct 31 01:38:03.694315 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:03.694260    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"
Oct 31 01:38:03.809446 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:03.809398    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8" podStartSLOduration=1.80938349 podStartE2EDuration="1.80938349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:03.806966652 +0000 UTC m=+2493.496849974" watchObservedRunningTime="2026-10-31 01:38:03.80938349 +0000 UTC m=+2493.499266665"
Oct 31 01:38:04.892821 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:04.892726    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:09.617461 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:09.617424    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:14.318874 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:14.318803    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:14.703609 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:14.703517    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"
Oct 31 01:38:19.766871 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:19.766834    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:22.332380 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:22.332343    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:23.392996 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:23.392962    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"]
Oct 31 01:38:23.497324 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:23.497288    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"]
Oct 31 01:38:23.801926 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:23.801892    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:24.119782 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:24.119695    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:27.054138 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:27.054104    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:28.394329 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.394289    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj" podUID="aae91494-86a8-468d-a16d-541b75623712" containerName="payload-processing" containerID="cri-o://58ef05694fe0b076863379ae00cb49bfbf04ae9be4f9813483dce30a0d3a8a9d" gracePeriod=25
Oct 31 01:38:28.497890 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.497857    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4" podUID="ff8f4f64-5ebd-4331-a385-91dcaed57849" containerName="payload-pre-processing" containerID="cri-o://34d1e7be4f464b86323505c2b864bf1365bb3d1f616550e4826cb4ae22c3315e" gracePeriod=25
Oct 31 01:38:28.654647 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.654583    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"
Oct 31 01:38:28.761537 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.761507    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"
Oct 31 01:38:28.793877 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.793851    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae91494-86a8-468d-a16d-541b75623712-kube-api-access-sjbgv\" (UniqueName: \"kubernetes.io/projected/aae91494-86a8-468d-a16d-541b75623712-kube-api-access-sjbgv\") pod \"aae91494-86a8-468d-a16d-541b75623712\" (UID: \"aae91494-86a8-468d-a16d-541b75623712\") "
Oct 31 01:38:28.794037 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.793889    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae91494-86a8-468d-a16d-541b75623712-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae91494-86a8-468d-a16d-541b75623712-plugins-config-volume\") pod \"aae91494-86a8-468d-a16d-541b75623712\" (UID: \"aae91494-86a8-468d-a16d-541b75623712\") "
Oct 31 01:38:28.795583 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.795547    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae91494-86a8-468d-a16d-541b75623712-plugins-config-volume" pod "aae91494-86a8-468d-a16d-541b75623712" (UID: "aae91494-86a8-468d-a16d-541b75623712"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:38:28.797687 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.797660    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae91494-86a8-468d-a16d-541b75623712-kube-api-access-sjbgv" pod "aae91494-86a8-468d-a16d-541b75623712" (UID: "aae91494-86a8-468d-a16d-541b75623712"). InnerVolumeSpecName "kube-api-access-sjbgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:28.848961 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.848934    4476 generic.go:356] "Generic (PLEG): container finished" podID="aae91494-86a8-468d-a16d-541b75623712" containerID="58ef05694fe0b076863379ae00cb49bfbf04ae9be4f9813483dce30a0d3a8a9d" exitCode=0
Oct 31 01:38:28.849095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.848977    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"
Oct 31 01:38:28.849095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.849017    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj" event={"ID":"aae91494-86a8-468d-a16d-541b75623712","Type":"ContainerDied","Data":"58ef05694fe0b076863379ae00cb49bfbf04ae9be4f9813483dce30a0d3a8a9d"}
Oct 31 01:38:28.849095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.849061    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj" event={"ID":"aae91494-86a8-468d-a16d-541b75623712","Type":"ContainerDied","Data":"39096616371d2282386098a73ba56d148b93ed52084689990d982ac47998fe92"}
Oct 31 01:38:28.849095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.849075    4476 scope.go:122] "RemoveContainer" containerID="58ef05694fe0b076863379ae00cb49bfbf04ae9be4f9813483dce30a0d3a8a9d"
Oct 31 01:38:28.850448 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.850424    4476 generic.go:356] "Generic (PLEG): container finished" podID="ff8f4f64-5ebd-4331-a385-91dcaed57849" containerID="34d1e7be4f464b86323505c2b864bf1365bb3d1f616550e4826cb4ae22c3315e" exitCode=0
Oct 31 01:38:28.850552 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.850471    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4" event={"ID":"ff8f4f64-5ebd-4331-a385-91dcaed57849","Type":"ContainerDied","Data":"34d1e7be4f464b86323505c2b864bf1365bb3d1f616550e4826cb4ae22c3315e"}
Oct 31 01:38:28.850552 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.850504    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4" event={"ID":"ff8f4f64-5ebd-4331-a385-91dcaed57849","Type":"ContainerDied","Data":"ad3ed21080566bad31c26c57dcbe79af1f6a6748f320b65db6a99e9dc038a2ed"}
Oct 31 01:38:28.850552 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.850516    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"
Oct 31 01:38:28.858849 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.858823    4476 scope.go:122] "RemoveContainer" containerID="58ef05694fe0b076863379ae00cb49bfbf04ae9be4f9813483dce30a0d3a8a9d"
Oct 31 01:38:28.859707 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:38:28.859679    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ef05694fe0b076863379ae00cb49bfbf04ae9be4f9813483dce30a0d3a8a9d\": container with ID starting with 58ef05694fe0b076863379ae00cb49bfbf04ae9be4f9813483dce30a0d3a8a9d not found: ID does not exist" containerID="58ef05694fe0b076863379ae00cb49bfbf04ae9be4f9813483dce30a0d3a8a9d"
Oct 31 01:38:28.859801 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.859714    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ef05694fe0b076863379ae00cb49bfbf04ae9be4f9813483dce30a0d3a8a9d"} err="failed to get container status \"58ef05694fe0b076863379ae00cb49bfbf04ae9be4f9813483dce30a0d3a8a9d\": rpc error: code = NotFound desc = could not find container \"58ef05694fe0b076863379ae00cb49bfbf04ae9be4f9813483dce30a0d3a8a9d\": container with ID starting with 58ef05694fe0b076863379ae00cb49bfbf04ae9be4f9813483dce30a0d3a8a9d not found: ID does not exist"
Oct 31 01:38:28.859801 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.859730    4476 scope.go:122] "RemoveContainer" containerID="34d1e7be4f464b86323505c2b864bf1365bb3d1f616550e4826cb4ae22c3315e"
Oct 31 01:38:28.868573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.868555    4476 scope.go:122] "RemoveContainer" containerID="34d1e7be4f464b86323505c2b864bf1365bb3d1f616550e4826cb4ae22c3315e"
Oct 31 01:38:28.868826 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:38:28.868810    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d1e7be4f464b86323505c2b864bf1365bb3d1f616550e4826cb4ae22c3315e\": container with ID starting with 34d1e7be4f464b86323505c2b864bf1365bb3d1f616550e4826cb4ae22c3315e not found: ID does not exist" containerID="34d1e7be4f464b86323505c2b864bf1365bb3d1f616550e4826cb4ae22c3315e"
Oct 31 01:38:28.868872 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.868832    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d1e7be4f464b86323505c2b864bf1365bb3d1f616550e4826cb4ae22c3315e"} err="failed to get container status \"34d1e7be4f464b86323505c2b864bf1365bb3d1f616550e4826cb4ae22c3315e\": rpc error: code = NotFound desc = could not find container \"34d1e7be4f464b86323505c2b864bf1365bb3d1f616550e4826cb4ae22c3315e\": container with ID starting with 34d1e7be4f464b86323505c2b864bf1365bb3d1f616550e4826cb4ae22c3315e not found: ID does not exist"
Oct 31 01:38:28.895159 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.895140    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff8f4f64-5ebd-4331-a385-91dcaed57849-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8f4f64-5ebd-4331-a385-91dcaed57849-plugins-config-volume\") pod \"ff8f4f64-5ebd-4331-a385-91dcaed57849\" (UID: \"ff8f4f64-5ebd-4331-a385-91dcaed57849\") "
Oct 31 01:38:28.895264 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.895169    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8f4f64-5ebd-4331-a385-91dcaed57849-kube-api-access-ddpgq\" (UniqueName: \"kubernetes.io/projected/ff8f4f64-5ebd-4331-a385-91dcaed57849-kube-api-access-ddpgq\") pod \"ff8f4f64-5ebd-4331-a385-91dcaed57849\" (UID: \"ff8f4f64-5ebd-4331-a385-91dcaed57849\") "
Oct 31 01:38:28.895461 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.895444    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8f4f64-5ebd-4331-a385-91dcaed57849-plugins-config-volume" pod "ff8f4f64-5ebd-4331-a385-91dcaed57849" (UID: "ff8f4f64-5ebd-4331-a385-91dcaed57849"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:38:28.895667 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.895652    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8f4f64-5ebd-4331-a385-91dcaed57849-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:28.895711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.895671    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjbgv\" (UniqueName: \"kubernetes.io/projected/aae91494-86a8-468d-a16d-541b75623712-kube-api-access-sjbgv\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:28.895711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.895682    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae91494-86a8-468d-a16d-541b75623712-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:28.897154 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.897138    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8f4f64-5ebd-4331-a385-91dcaed57849-kube-api-access-ddpgq" pod "ff8f4f64-5ebd-4331-a385-91dcaed57849" (UID: "ff8f4f64-5ebd-4331-a385-91dcaed57849"). InnerVolumeSpecName "kube-api-access-ddpgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:28.996511 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:28.996445    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddpgq\" (UniqueName: \"kubernetes.io/projected/ff8f4f64-5ebd-4331-a385-91dcaed57849-kube-api-access-ddpgq\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:29.019752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:29.019724    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"]
Oct 31 01:38:29.024002 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:29.023975    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f0818-55fdff8547-jpvnj"]
Oct 31 01:38:29.180655 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:38:29.180617    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:38:29.270563 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:29.270495    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"]
Oct 31 01:38:29.274602 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:29.274580    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3f0818-5c5c6677c5-qmpq4"]
Oct 31 01:38:30.455215 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:30.455159    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae91494-86a8-468d-a16d-541b75623712" path="/var/lib/kubelet/pods/aae91494-86a8-468d-a16d-541b75623712/volumes"
Oct 31 01:38:30.455745 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:30.455724    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff8f4f64-5ebd-4331-a385-91dcaed57849" path="/var/lib/kubelet/pods/ff8f4f64-5ebd-4331-a385-91dcaed57849/volumes"
Oct 31 01:38:32.649133 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:32.649099    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:38.172819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:38.172786    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:38.688915 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:38.688884    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:38.869311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:38.869281    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:39.048765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:39.048734    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:39.766262 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:39.766233    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:40.012087 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:40.012051    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:40.195423 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:40.195392    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:40.808809 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:40.808773    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:45.166401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:45.166368    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:46.946936 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:46.946905    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:47.154975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:47.154941    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:47.419340 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:47.419307    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:50.472837 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:50.472778    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:54.295501 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:54.295469    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:54.817984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:54.817951    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:56.901599 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:56.901565    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:57.421608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:57.421579    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:57.604819 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:57.604789    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:58.091507 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:58.091479    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:58.450157 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:58.450079    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:58.676295 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:58.676259    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:58.947645 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:58.947604    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:59.182254 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:38:59.182222    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:38:59.520408 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:59.520382    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:59.860707 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:38:59.860675    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:00.155008 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:00.154934    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:00.343078 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:00.343045    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:00.941239 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:00.941210    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:05.739510 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:05.739477    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:06.117329 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:06.117291    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:06.494967 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:06.494881    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:06.876030 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:06.875994    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:07.094356 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:07.094330    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:09.358647 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:09.358609    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:09.674336 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:09.674257    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"]
Oct 31 01:39:09.674513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:09.674489    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv" podUID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerName="istio-proxy" containerID="cri-o://a6dd4c634b15e3db4891204e05c083111679532de1e106784bf77eb3b3a453a2" gracePeriod=30
Oct 31 01:39:09.693829 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:09.693799    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:10.174093 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:10.174063    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:11.121297 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:11.121205    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:11.229979 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:11.229948    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:39:11.230154 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:11.229996    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv" podUID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:39:11.425435 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:11.425358    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:12.171291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:12.171252    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:12.734471 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:12.734440    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:14.925220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.925195    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:39:14.986287 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.986228    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-podinfo\") pod \"185b3025-1c9e-4c04-9beb-a21c93225c0f\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") "
Oct 31 01:39:14.986287 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.986267    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-envoy\") pod \"185b3025-1c9e-4c04-9beb-a21c93225c0f\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") "
Oct 31 01:39:14.986472 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.986299    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-credential-socket\") pod \"185b3025-1c9e-4c04-9beb-a21c93225c0f\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") "
Oct 31 01:39:14.986472 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.986324    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185b3025-1c9e-4c04-9beb-a21c93225c0f-kube-api-access-46njq\" (UniqueName: \"kubernetes.io/projected/185b3025-1c9e-4c04-9beb-a21c93225c0f-kube-api-access-46njq\") pod \"185b3025-1c9e-4c04-9beb-a21c93225c0f\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") "
Oct 31 01:39:14.986472 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.986388    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-data\") pod \"185b3025-1c9e-4c04-9beb-a21c93225c0f\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") "
Oct 31 01:39:14.986472 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.986412    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185b3025-1c9e-4c04-9beb-a21c93225c0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b3025-1c9e-4c04-9beb-a21c93225c0f-istiod-ca-cert\") pod \"185b3025-1c9e-4c04-9beb-a21c93225c0f\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") "
Oct 31 01:39:14.986688 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.986593    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-workload-socket\") pod \"185b3025-1c9e-4c04-9beb-a21c93225c0f\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") "
Oct 31 01:39:14.986688 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.986658    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-data" pod "185b3025-1c9e-4c04-9beb-a21c93225c0f" (UID: "185b3025-1c9e-4c04-9beb-a21c93225c0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:39:14.986791 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.986677    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-credential-socket" pod "185b3025-1c9e-4c04-9beb-a21c93225c0f" (UID: "185b3025-1c9e-4c04-9beb-a21c93225c0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:39:14.986791 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.986693    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-workload-certs\") pod \"185b3025-1c9e-4c04-9beb-a21c93225c0f\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") "
Oct 31 01:39:14.986791 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.986734    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185b3025-1c9e-4c04-9beb-a21c93225c0f-istiod-ca-cert" pod "185b3025-1c9e-4c04-9beb-a21c93225c0f" (UID: "185b3025-1c9e-4c04-9beb-a21c93225c0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:39:14.986946 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.986865    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-token\" (UniqueName: \"kubernetes.io/projected/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-token\") pod \"185b3025-1c9e-4c04-9beb-a21c93225c0f\" (UID: \"185b3025-1c9e-4c04-9beb-a21c93225c0f\") "
Oct 31 01:39:14.986946 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.986913    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-workload-certs" pod "185b3025-1c9e-4c04-9beb-a21c93225c0f" (UID: "185b3025-1c9e-4c04-9beb-a21c93225c0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:39:14.986946 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.986928    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-workload-socket" pod "185b3025-1c9e-4c04-9beb-a21c93225c0f" (UID: "185b3025-1c9e-4c04-9beb-a21c93225c0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:39:14.987333 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.987315    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:39:14.987421 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.987337    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:39:14.987421 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.987351    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b3025-1c9e-4c04-9beb-a21c93225c0f-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:39:14.987421 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.987366    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:39:14.987421 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.987378    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:39:14.988757 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.988730    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185b3025-1c9e-4c04-9beb-a21c93225c0f-kube-api-access-46njq" pod "185b3025-1c9e-4c04-9beb-a21c93225c0f" (UID: "185b3025-1c9e-4c04-9beb-a21c93225c0f"). InnerVolumeSpecName "kube-api-access-46njq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:39:14.988957 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.988938    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-token" pod "185b3025-1c9e-4c04-9beb-a21c93225c0f" (UID: "185b3025-1c9e-4c04-9beb-a21c93225c0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:39:14.989050 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.989032    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-podinfo" pod "185b3025-1c9e-4c04-9beb-a21c93225c0f" (UID: "185b3025-1c9e-4c04-9beb-a21c93225c0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:39:14.989091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:14.989034    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-envoy" pod "185b3025-1c9e-4c04-9beb-a21c93225c0f" (UID: "185b3025-1c9e-4c04-9beb-a21c93225c0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:39:15.088803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.088777    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:39:15.088803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.088799    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:39:15.089001 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.088809    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b3025-1c9e-4c04-9beb-a21c93225c0f-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:39:15.089001 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.088818    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46njq\" (UniqueName: \"kubernetes.io/projected/185b3025-1c9e-4c04-9beb-a21c93225c0f-kube-api-access-46njq\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:39:15.137374 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.137341    4476 generic.go:356] "Generic (PLEG): container finished" podID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerID="a6dd4c634b15e3db4891204e05c083111679532de1e106784bf77eb3b3a453a2" exitCode=0
Oct 31 01:39:15.137544 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.137399    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"
Oct 31 01:39:15.137544 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.137414    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv" event={"ID":"185b3025-1c9e-4c04-9beb-a21c93225c0f","Type":"ContainerDied","Data":"a6dd4c634b15e3db4891204e05c083111679532de1e106784bf77eb3b3a453a2"}
Oct 31 01:39:15.137544 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.137465    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv" event={"ID":"185b3025-1c9e-4c04-9beb-a21c93225c0f","Type":"ContainerDied","Data":"6d9603048c06be9b14299363afc4465669b35b6da4436cb750d208e5cc0b758a"}
Oct 31 01:39:15.137544 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.137498    4476 scope.go:122] "RemoveContainer" containerID="a6dd4c634b15e3db4891204e05c083111679532de1e106784bf77eb3b3a453a2"
Oct 31 01:39:15.147883 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.147860    4476 scope.go:122] "RemoveContainer" containerID="982d584be2c8f70db2d53a525d56dae4c1a02640f0084d0445ecdee832916e4e"
Oct 31 01:39:15.156637 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.156620    4476 scope.go:122] "RemoveContainer" containerID="a6dd4c634b15e3db4891204e05c083111679532de1e106784bf77eb3b3a453a2"
Oct 31 01:39:15.156908 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:39:15.156883    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6dd4c634b15e3db4891204e05c083111679532de1e106784bf77eb3b3a453a2\": container with ID starting with a6dd4c634b15e3db4891204e05c083111679532de1e106784bf77eb3b3a453a2 not found: ID does not exist" containerID="a6dd4c634b15e3db4891204e05c083111679532de1e106784bf77eb3b3a453a2"
Oct 31 01:39:15.156953 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.156919    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6dd4c634b15e3db4891204e05c083111679532de1e106784bf77eb3b3a453a2"} err="failed to get container status \"a6dd4c634b15e3db4891204e05c083111679532de1e106784bf77eb3b3a453a2\": rpc error: code = NotFound desc = could not find container \"a6dd4c634b15e3db4891204e05c083111679532de1e106784bf77eb3b3a453a2\": container with ID starting with a6dd4c634b15e3db4891204e05c083111679532de1e106784bf77eb3b3a453a2 not found: ID does not exist"
Oct 31 01:39:15.156953 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.156944    4476 scope.go:122] "RemoveContainer" containerID="982d584be2c8f70db2d53a525d56dae4c1a02640f0084d0445ecdee832916e4e"
Oct 31 01:39:15.157221 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:39:15.157189    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"982d584be2c8f70db2d53a525d56dae4c1a02640f0084d0445ecdee832916e4e\": container with ID starting with 982d584be2c8f70db2d53a525d56dae4c1a02640f0084d0445ecdee832916e4e not found: ID does not exist" containerID="982d584be2c8f70db2d53a525d56dae4c1a02640f0084d0445ecdee832916e4e"
Oct 31 01:39:15.157281 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.157228    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"982d584be2c8f70db2d53a525d56dae4c1a02640f0084d0445ecdee832916e4e"} err="failed to get container status \"982d584be2c8f70db2d53a525d56dae4c1a02640f0084d0445ecdee832916e4e\": rpc error: code = NotFound desc = could not find container \"982d584be2c8f70db2d53a525d56dae4c1a02640f0084d0445ecdee832916e4e\": container with ID starting with 982d584be2c8f70db2d53a525d56dae4c1a02640f0084d0445ecdee832916e4e not found: ID does not exist"
Oct 31 01:39:15.304927 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.304898    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"]
Oct 31 01:39:15.309342 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:15.309322    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6c7db9-openshift-default-69bd847677-f6slv"]
Oct 31 01:39:16.454013 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:16.453981    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="185b3025-1c9e-4c04-9beb-a21c93225c0f" path="/var/lib/kubelet/pods/185b3025-1c9e-4c04-9beb-a21c93225c0f/volumes"
Oct 31 01:39:17.142927 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:17.142893    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:22.316431 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:22.316381    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:27.737192 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:27.733808    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:29.182061 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:39:29.182027    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:39:31.396766 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:31.396685    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:31.848050 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:31.848015    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:33.857871 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:33.857832    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:34.371910 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:34.371872    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:34.559088 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:34.559049    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:35.414071 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:35.413999    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:35.678335 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:35.678253    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:35.871080 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:35.871042    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:36.378065 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:36.378031    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:39.258092 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:39.258059    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:39.676891 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:39.676861    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:41.105954 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:41.105914    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:43.098188 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:43.098151    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:43.390667 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:43.390584    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:47.711199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:47.711144    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:52.607573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:52.607536    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:53.457224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:53.457192    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:57.531681 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:39:57.531638    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:59.181396 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:39:59.181360    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:40:03.362669 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:03.362632    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:03.869275 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:03.869243    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:04.056394 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:04.056362    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:04.940549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:04.940519    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:05.135805 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:05.135769    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:05.194353 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:05.194272    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vl9ww_c0881ec8-1f45-459a-b40c-29e2d2246ea0/payload-processing/0.log"
Oct 31 01:40:05.319849 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:05.319815    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:05.450873 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:05.450805    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vl9ww_c0881ec8-1f45-459a-b40c-29e2d2246ea0/payload-processing/0.log"
Oct 31 01:40:05.846504 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:05.846471    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:10.143391 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:10.143361    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:10.338843 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:10.338791    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:11.677135 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:11.677106    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:11.887374 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:11.887343    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:11.971335 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:11.971255    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"]
Oct 31 01:40:11.971494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:11.971471    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8" podUID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" containerName="istio-proxy" containerID="cri-o://3aac0e9d4be79aac157683651330fec8738f2ff408137294a5635e9a32793cf2" gracePeriod=30
Oct 31 01:40:12.099284 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:12.099249    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:13.579569 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:13.579534    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:14.279552 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:14.279518    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:14.476810 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:14.476761    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:15.676550 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:15.676508    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:15.876430 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:15.876399    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:16.062250 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:16.062217    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:17.241944 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.241887    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:40:17.274125 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.274097    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-credential-socket\") pod \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") "
Oct 31 01:40:17.274255 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.274165    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-kube-api-access-fsk72\" (UniqueName: \"kubernetes.io/projected/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-kube-api-access-fsk72\") pod \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") "
Oct 31 01:40:17.274255 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.274247    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-podinfo\") pod \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") "
Oct 31 01:40:17.274345 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.274280    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-workload-certs\") pod \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") "
Oct 31 01:40:17.274345 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.274316    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-envoy\") pod \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") "
Oct 31 01:40:17.274443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.274355    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istiod-ca-cert\") pod \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") "
Oct 31 01:40:17.274443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.274396    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-credential-socket" pod "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" (UID: "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:17.274443 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.274403    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-workload-socket\") pod \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") "
Oct 31 01:40:17.274595 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.274490    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-data\") pod \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") "
Oct 31 01:40:17.274654 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.274596    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-token\" (UniqueName: \"kubernetes.io/projected/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-token\") pod \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\" (UID: \"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa\") "
Oct 31 01:40:17.274654 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.274599    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-workload-certs" pod "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" (UID: "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:17.274780 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.274723    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istiod-ca-cert" pod "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" (UID: "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:17.274780 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.274722    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-workload-socket" pod "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" (UID: "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:17.274885 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.274827    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-data" pod "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" (UID: "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:17.275441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.275418    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.275545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.275446    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.275545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.275462    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.275545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.275477    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.275545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.275491    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.276542 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.276521    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-envoy" pod "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" (UID: "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:17.276645 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.276616    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-podinfo" pod "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" (UID: "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:40:17.276695 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.276648    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-kube-api-access-fsk72" pod "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" (UID: "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa"). InnerVolumeSpecName "kube-api-access-fsk72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:17.276783 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.276769    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-token" pod "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" (UID: "aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:17.376102 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.376072    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsk72\" (UniqueName: \"kubernetes.io/projected/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-kube-api-access-fsk72\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.376102 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.376098    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.376282 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.376109    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.376282 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.376118    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.484081 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.484052    4476 generic.go:356] "Generic (PLEG): container finished" podID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" containerID="3aac0e9d4be79aac157683651330fec8738f2ff408137294a5635e9a32793cf2" exitCode=0
Oct 31 01:40:17.484257 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.484103    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"
Oct 31 01:40:17.484257 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.484134    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8" event={"ID":"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa","Type":"ContainerDied","Data":"3aac0e9d4be79aac157683651330fec8738f2ff408137294a5635e9a32793cf2"}
Oct 31 01:40:17.484257 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.484203    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8" event={"ID":"aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa","Type":"ContainerDied","Data":"03cb53e0ff4e926704482e35930a996de4f3cb9057344b24a5b1be180a0df991"}
Oct 31 01:40:17.484257 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.484231    4476 scope.go:122] "RemoveContainer" containerID="3aac0e9d4be79aac157683651330fec8738f2ff408137294a5635e9a32793cf2"
Oct 31 01:40:17.497351 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.497331    4476 scope.go:122] "RemoveContainer" containerID="27663cef3d053c2d5799ff5158d38f355fad2f7258eca875cca776395de2d772"
Oct 31 01:40:17.506329 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.506314    4476 scope.go:122] "RemoveContainer" containerID="3aac0e9d4be79aac157683651330fec8738f2ff408137294a5635e9a32793cf2"
Oct 31 01:40:17.506567 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:40:17.506552    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aac0e9d4be79aac157683651330fec8738f2ff408137294a5635e9a32793cf2\": container with ID starting with 3aac0e9d4be79aac157683651330fec8738f2ff408137294a5635e9a32793cf2 not found: ID does not exist" containerID="3aac0e9d4be79aac157683651330fec8738f2ff408137294a5635e9a32793cf2"
Oct 31 01:40:17.506620 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.506576    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aac0e9d4be79aac157683651330fec8738f2ff408137294a5635e9a32793cf2"} err="failed to get container status \"3aac0e9d4be79aac157683651330fec8738f2ff408137294a5635e9a32793cf2\": rpc error: code = NotFound desc = could not find container \"3aac0e9d4be79aac157683651330fec8738f2ff408137294a5635e9a32793cf2\": container with ID starting with 3aac0e9d4be79aac157683651330fec8738f2ff408137294a5635e9a32793cf2 not found: ID does not exist"
Oct 31 01:40:17.506620 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.506592    4476 scope.go:122] "RemoveContainer" containerID="27663cef3d053c2d5799ff5158d38f355fad2f7258eca875cca776395de2d772"
Oct 31 01:40:17.506808 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:40:17.506789    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27663cef3d053c2d5799ff5158d38f355fad2f7258eca875cca776395de2d772\": container with ID starting with 27663cef3d053c2d5799ff5158d38f355fad2f7258eca875cca776395de2d772 not found: ID does not exist" containerID="27663cef3d053c2d5799ff5158d38f355fad2f7258eca875cca776395de2d772"
Oct 31 01:40:17.506844 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.506816    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27663cef3d053c2d5799ff5158d38f355fad2f7258eca875cca776395de2d772"} err="failed to get container status \"27663cef3d053c2d5799ff5158d38f355fad2f7258eca875cca776395de2d772\": rpc error: code = NotFound desc = could not find container \"27663cef3d053c2d5799ff5158d38f355fad2f7258eca875cca776395de2d772\": container with ID starting with 27663cef3d053c2d5799ff5158d38f355fad2f7258eca875cca776395de2d772 not found: ID does not exist"
Oct 31 01:40:17.653118 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.653090    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"]
Oct 31 01:40:17.659344 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:17.659312    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-634135-openshift-default-7f7b88d49-xc6n8"]
Oct 31 01:40:18.095626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:18.095597    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:18.307374 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:18.307347    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:18.453989 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:18.453916    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" path="/var/lib/kubelet/pods/aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa/volumes"
Oct 31 01:40:18.525458 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:18.525429    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:20.976971 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:20.976937    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:21.223464 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:21.223436    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:22.287884 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:22.287857    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:22.503345 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:22.503308    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:22.789182 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:22.789148    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:23.361437 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:23.361411    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:23.579957 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:23.579930    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:25.183266 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.183233    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"]
Oct 31 01:40:25.183657 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.183465    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4" podUID="bf5cc350-c137-491c-94e2-efafdfb82a0f" containerName="maas-api" containerID="cri-o://fc72b336c7c781567bde23bbdf156f83dfc550a1df7fafa17ce4a0fc50882a3c" gracePeriod=30
Oct 31 01:40:25.274396 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.274361    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4" podUID="bf5cc350-c137-491c-94e2-efafdfb82a0f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": dial tcp 10.133.0.112:8443: connect: connection refused"
Oct 31 01:40:25.340650 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.340620    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:25.445215 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.445130    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:40:25.529262 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.529231    4476 generic.go:356] "Generic (PLEG): container finished" podID="bf5cc350-c137-491c-94e2-efafdfb82a0f" containerID="fc72b336c7c781567bde23bbdf156f83dfc550a1df7fafa17ce4a0fc50882a3c" exitCode=0
Oct 31 01:40:25.529407 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.529274    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"
Oct 31 01:40:25.529407 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.529302    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4" event={"ID":"bf5cc350-c137-491c-94e2-efafdfb82a0f","Type":"ContainerDied","Data":"fc72b336c7c781567bde23bbdf156f83dfc550a1df7fafa17ce4a0fc50882a3c"}
Oct 31 01:40:25.529407 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.529347    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4" event={"ID":"bf5cc350-c137-491c-94e2-efafdfb82a0f","Type":"ContainerDied","Data":"1a99f77b3deb1847bb7524670b4426d7df90204422d2c2f977fe5164d47f3e0d"}
Oct 31 01:40:25.529407 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.529369    4476 scope.go:122] "RemoveContainer" containerID="fc72b336c7c781567bde23bbdf156f83dfc550a1df7fafa17ce4a0fc50882a3c"
Oct 31 01:40:25.538588 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.538569    4476 scope.go:122] "RemoveContainer" containerID="fc72b336c7c781567bde23bbdf156f83dfc550a1df7fafa17ce4a0fc50882a3c"
Oct 31 01:40:25.538845 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:40:25.538827    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc72b336c7c781567bde23bbdf156f83dfc550a1df7fafa17ce4a0fc50882a3c\": container with ID starting with fc72b336c7c781567bde23bbdf156f83dfc550a1df7fafa17ce4a0fc50882a3c not found: ID does not exist" containerID="fc72b336c7c781567bde23bbdf156f83dfc550a1df7fafa17ce4a0fc50882a3c"
Oct 31 01:40:25.538912 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.538858    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc72b336c7c781567bde23bbdf156f83dfc550a1df7fafa17ce4a0fc50882a3c"} err="failed to get container status \"fc72b336c7c781567bde23bbdf156f83dfc550a1df7fafa17ce4a0fc50882a3c\": rpc error: code = NotFound desc = could not find container \"fc72b336c7c781567bde23bbdf156f83dfc550a1df7fafa17ce4a0fc50882a3c\": container with ID starting with fc72b336c7c781567bde23bbdf156f83dfc550a1df7fafa17ce4a0fc50882a3c not found: ID does not exist"
Oct 31 01:40:25.571097 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.571070    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5cc350-c137-491c-94e2-efafdfb82a0f-kube-api-access-d6lln\" (UniqueName: \"kubernetes.io/projected/bf5cc350-c137-491c-94e2-efafdfb82a0f-kube-api-access-d6lln\") pod \"bf5cc350-c137-491c-94e2-efafdfb82a0f\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") "
Oct 31 01:40:25.571294 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.571146    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf5cc350-c137-491c-94e2-efafdfb82a0f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5cc350-c137-491c-94e2-efafdfb82a0f-openshift-service-ca\") pod \"bf5cc350-c137-491c-94e2-efafdfb82a0f\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") "
Oct 31 01:40:25.571294 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.571222    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-metrics-certs\") pod \"bf5cc350-c137-491c-94e2-efafdfb82a0f\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") "
Oct 31 01:40:25.571294 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.571263    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-maas-api-tls\") pod \"bf5cc350-c137-491c-94e2-efafdfb82a0f\" (UID: \"bf5cc350-c137-491c-94e2-efafdfb82a0f\") "
Oct 31 01:40:25.572751 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.571799    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5cc350-c137-491c-94e2-efafdfb82a0f-openshift-service-ca" pod "bf5cc350-c137-491c-94e2-efafdfb82a0f" (UID: "bf5cc350-c137-491c-94e2-efafdfb82a0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:25.572751 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.572237    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5cc350-c137-491c-94e2-efafdfb82a0f-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:25.575107 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.575086    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-maas-api-tls" pod "bf5cc350-c137-491c-94e2-efafdfb82a0f" (UID: "bf5cc350-c137-491c-94e2-efafdfb82a0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:40:25.585741 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.585714    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-metrics-certs" pod "bf5cc350-c137-491c-94e2-efafdfb82a0f" (UID: "bf5cc350-c137-491c-94e2-efafdfb82a0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:40:25.585862 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.585839    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5cc350-c137-491c-94e2-efafdfb82a0f-kube-api-access-d6lln" pod "bf5cc350-c137-491c-94e2-efafdfb82a0f" (UID: "bf5cc350-c137-491c-94e2-efafdfb82a0f"). InnerVolumeSpecName "kube-api-access-d6lln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:25.673756 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.673731    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:25.673756 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.673758    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf5cc350-c137-491c-94e2-efafdfb82a0f-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:25.673913 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.673773    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6lln\" (UniqueName: \"kubernetes.io/projected/bf5cc350-c137-491c-94e2-efafdfb82a0f-kube-api-access-d6lln\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:25.962874 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.962841    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:25.963894 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.963874    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"]
Oct 31 01:40:25.974437 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:25.974417    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f0818-544c98b547-zzrj4"]
Oct 31 01:40:26.185628 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:26.185556    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:26.454972 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:26.454894    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf5cc350-c137-491c-94e2-efafdfb82a0f" path="/var/lib/kubelet/pods/bf5cc350-c137-491c-94e2-efafdfb82a0f/volumes"
Oct 31 01:40:27.066612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:27.066579    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:27.424035 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:27.423957    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:27.744067 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:27.743984    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:28.055467 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:28.055436    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:28.359971 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:28.359848    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:29.180648 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:40:29.180610    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:40:31.389168 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:31.389091    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:31.603914 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:31.603886    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:31.998771 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:31.998732    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:32.954851 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:32.954812    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:34.403484 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:34.403448    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:34.610895 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:34.610855    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:34.820404 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:34.820372    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:36.315647 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:36.315618    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:38.060732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:38.060698    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:39.651029 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:39.650995    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:41.413200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:41.413091    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:41.619023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:41.618996    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:43.409328 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:43.409300    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:44.630161 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:44.630129    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:44.825851 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:44.825823    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:49.776304 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:49.776265    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:54.375678 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:54.375639    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:59.079065 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:40:59.079026    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:59.180885 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:40:59.180852    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:41:03.418105 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418069    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"]
Oct 31 01:41:03.418610 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418594    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ff8f4f64-5ebd-4331-a385-91dcaed57849" containerName="payload-pre-processing"
Oct 31 01:41:03.418654 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418611    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8f4f64-5ebd-4331-a385-91dcaed57849" containerName="payload-pre-processing"
Oct 31 01:41:03.418654 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418634    4476 cpu_manager.go:414] "containerMap: removing container" podUID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" containerName="istio-proxy"
Oct 31 01:41:03.418654 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418640    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" containerName="istio-proxy"
Oct 31 01:41:03.418654 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418649    4476 cpu_manager.go:414] "containerMap: removing container" podUID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerName="istio-proxy"
Oct 31 01:41:03.418654 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418655    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerName="istio-proxy"
Oct 31 01:41:03.418808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418666    4476 cpu_manager.go:414] "containerMap: removing container" podUID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerName="istio-proxy"
Oct 31 01:41:03.418808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418670    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerName="istio-proxy"
Oct 31 01:41:03.418808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418676    4476 cpu_manager.go:414] "containerMap: removing container" podUID="bf5cc350-c137-491c-94e2-efafdfb82a0f" containerName="maas-api"
Oct 31 01:41:03.418808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418681    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5cc350-c137-491c-94e2-efafdfb82a0f" containerName="maas-api"
Oct 31 01:41:03.418808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418693    4476 cpu_manager.go:414] "containerMap: removing container" podUID="aae91494-86a8-468d-a16d-541b75623712" containerName="payload-processing"
Oct 31 01:41:03.418808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418699    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae91494-86a8-468d-a16d-541b75623712" containerName="payload-processing"
Oct 31 01:41:03.418808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418768    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerName="istio-proxy"
Oct 31 01:41:03.418808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418776    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="185b3025-1c9e-4c04-9beb-a21c93225c0f" containerName="istio-proxy"
Oct 31 01:41:03.418808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418782    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5cc350-c137-491c-94e2-efafdfb82a0f" containerName="maas-api"
Oct 31 01:41:03.418808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418789    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" containerName="istio-proxy"
Oct 31 01:41:03.418808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418794    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae91494-86a8-468d-a16d-541b75623712" containerName="payload-processing"
Oct 31 01:41:03.418808 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.418801    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8f4f64-5ebd-4331-a385-91dcaed57849" containerName="payload-pre-processing"
Oct 31 01:41:03.422073 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.422059    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:03.525419 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.525387    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-25b9ff-serving-cert\""
Oct 31 01:41:03.528066 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.528037    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"]
Oct 31 01:41:03.560222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.560164    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-maas-api-tls\") pod \"maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:03.560368 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.560246    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cd6adf-c72d-497e-bed0-c64642bb8dea-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:03.560419 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.560366    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74xtz\" (UniqueName: \"kubernetes.io/projected/50cd6adf-c72d-497e-bed0-c64642bb8dea-kube-api-access-74xtz\") pod \"maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:03.560419 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.560406    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-metrics-certs\") pod \"maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:03.660110 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.660078    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:03.661489 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.661466    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-maas-api-tls\") pod \"maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:03.661650 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.661511    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cd6adf-c72d-497e-bed0-c64642bb8dea-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:03.661650 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.661571    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74xtz\" (UniqueName: \"kubernetes.io/projected/50cd6adf-c72d-497e-bed0-c64642bb8dea-kube-api-access-74xtz\") pod \"maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:03.661650 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.661597    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-metrics-certs\") pod \"maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:03.661650 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:41:03.661633    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-serving-cert: secret "maas-api-e2e-praxis-a-25b9ff-serving-cert" not found
Oct 31 01:41:03.661891 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:41:03.661716    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-maas-api-tls podName:50cd6adf-c72d-497e-bed0-c64642bb8dea nodeName:}" failed. No retries permitted until 2026-10-31 01:41:04.161694137 +0000 UTC m=+2673.851577242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-maas-api-tls") pod "maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr" (UID: "50cd6adf-c72d-497e-bed0-c64642bb8dea") : secret "maas-api-e2e-praxis-a-25b9ff-serving-cert" not found
Oct 31 01:41:03.662284 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.662265    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cd6adf-c72d-497e-bed0-c64642bb8dea-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:03.665032 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.665008    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-metrics-certs\") pod \"maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:03.718489 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.718416    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74xtz\" (UniqueName: \"kubernetes.io/projected/50cd6adf-c72d-497e-bed0-c64642bb8dea-kube-api-access-74xtz\") pod \"maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:03.834870 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:03.834835    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:04.165855 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:04.165819    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-maas-api-tls\") pod \"maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:04.168368 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:04.168349    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-maas-api-tls\") pod \"maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:04.335235 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:04.335202    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:04.707458 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:04.707434    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"]
Oct 31 01:41:04.709137 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:41:04.709112    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cd6adf_c72d_497e_bed0_c64642bb8dea.slice/crio-af263559aaeb3667adec4c9a67317086286d133c8f916a30be868d6f100ad716 WatchSource:0}: Error finding container af263559aaeb3667adec4c9a67317086286d133c8f916a30be868d6f100ad716: Status 404 returned error can't find the container with id af263559aaeb3667adec4c9a67317086286d133c8f916a30be868d6f100ad716
Oct 31 01:41:04.741933 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:04.741902    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr" event={"ID":"50cd6adf-c72d-497e-bed0-c64642bb8dea","Type":"ContainerStarted","Data":"af263559aaeb3667adec4c9a67317086286d133c8f916a30be868d6f100ad716"}
Oct 31 01:41:05.749137 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:05.749093    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr" event={"ID":"50cd6adf-c72d-497e-bed0-c64642bb8dea","Type":"ContainerStarted","Data":"ff833358e023aeeef54c4ba4e559cd5886ede3dbdc3c08965b46faa33466437c"}
Oct 31 01:41:05.749671 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:05.749298    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:05.863225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:05.863127    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr" podStartSLOduration=1.996319416 podStartE2EDuration="2.863114937s" totalImagesPullingTime="866.795521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:05.861465419 +0000 UTC m=+2675.551348547" watchObservedRunningTime="2026-10-31 01:41:05.863114937 +0000 UTC m=+2675.552998103"
Oct 31 01:41:08.500474 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:08.500439    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:11.759449 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:11.759418    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:41:13.119676 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:13.119637    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:13.303538 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:13.303506    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:13.490567 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:13.490468    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:17.997601 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:17.997536    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:20.481094 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:20.481064    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:22.395033 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.395005    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"]
Oct 31 01:41:22.395511 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.395497    4476 cpu_manager.go:414] "containerMap: removing container" podUID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" containerName="istio-proxy"
Oct 31 01:41:22.395560 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.395511    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" containerName="istio-proxy"
Oct 31 01:41:22.395618 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.395609    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa12d5ea-cb6d-4c40-9f13-ed3c0b717bfa" containerName="istio-proxy"
Oct 31 01:41:22.398917 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.398902    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:22.452097 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.452075    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-66100b-serving-cert\""
Oct 31 01:41:22.456124 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.456104    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"]
Oct 31 01:41:22.462313 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.462264    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-maas-api-tls\") pod \"maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:22.462404 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.462332    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ee38a2-7a31-49ca-b903-af33df249cfa-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:22.462404 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.462365    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74hpl\" (UniqueName: \"kubernetes.io/projected/89ee38a2-7a31-49ca-b903-af33df249cfa-kube-api-access-74hpl\") pod \"maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:22.462475 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.462430    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-metrics-certs\") pod \"maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:22.563216 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.563167    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ee38a2-7a31-49ca-b903-af33df249cfa-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:22.563407 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.563229    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74hpl\" (UniqueName: \"kubernetes.io/projected/89ee38a2-7a31-49ca-b903-af33df249cfa-kube-api-access-74hpl\") pod \"maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:22.563407 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.563278    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-metrics-certs\") pod \"maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:22.563552 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.563433    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-maas-api-tls\") pod \"maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:22.563614 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:41:22.563558    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-serving-cert: secret "maas-api-e2e-praxis-a-66100b-serving-cert" not found
Oct 31 01:41:22.563676 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:41:22.563626    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-maas-api-tls podName:89ee38a2-7a31-49ca-b903-af33df249cfa nodeName:}" failed. No retries permitted until 2026-10-31 01:41:23.063605955 +0000 UTC m=+2692.753489060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-maas-api-tls") pod "maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld" (UID: "89ee38a2-7a31-49ca-b903-af33df249cfa") : secret "maas-api-e2e-praxis-a-66100b-serving-cert" not found
Oct 31 01:41:22.563977 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.563948    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ee38a2-7a31-49ca-b903-af33df249cfa-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:22.566373 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.566337    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-metrics-certs\") pod \"maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:22.622091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.622060    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74hpl\" (UniqueName: \"kubernetes.io/projected/89ee38a2-7a31-49ca-b903-af33df249cfa-kube-api-access-74hpl\") pod \"maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:22.630207 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.630162    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:22.818753 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.818721    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:22.994602 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:22.994574    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:23.071122 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:23.071033    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-maas-api-tls\") pod \"maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:23.073640 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:23.073614    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-maas-api-tls\") pod \"maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:23.312208 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:23.312160    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:23.685265 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:23.685239    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"]
Oct 31 01:41:23.687399 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:41:23.687370    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ee38a2_7a31_49ca_b903_af33df249cfa.slice/crio-8cf7c213ea585af203e04a1837d602383550173a4a788e1719a63aaef722b70f WatchSource:0}: Error finding container 8cf7c213ea585af203e04a1837d602383550173a4a788e1719a63aaef722b70f: Status 404 returned error can't find the container with id 8cf7c213ea585af203e04a1837d602383550173a4a788e1719a63aaef722b70f
Oct 31 01:41:23.688641 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:23.688623    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:41:23.845373 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:23.845339    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld" event={"ID":"89ee38a2-7a31-49ca-b903-af33df249cfa","Type":"ContainerStarted","Data":"8cf7c213ea585af203e04a1837d602383550173a4a788e1719a63aaef722b70f"}
Oct 31 01:41:24.852740 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:24.852699    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld" event={"ID":"89ee38a2-7a31-49ca-b903-af33df249cfa","Type":"ContainerStarted","Data":"b1a13934fb0676463765e397422ff8ec57ef71ecca0049b3e06201894810fdbb"}
Oct 31 01:41:24.853166 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:24.852765    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:24.970033 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:24.969982    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld" podStartSLOduration=2.146264381 podStartE2EDuration="2.96996869s" totalImagesPullingTime="823.704309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:24.966952152 +0000 UTC m=+2694.656835278" watchObservedRunningTime="2026-10-31 01:41:24.96996869 +0000 UTC m=+2694.659851817"
Oct 31 01:41:26.740748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:26.740714    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"]
Oct 31 01:41:26.741267 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:26.740943    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9" podUID="c8a86b81-5e45-4c08-9d7c-9f1b8f11a201" containerName="maas-api" containerID="cri-o://a9169e3dacedcad9a34ece18477f80371b25604a1e192a888ba8676175e7e133" gracePeriod=30
Oct 31 01:41:26.864829 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:26.864800    4476 generic.go:356] "Generic (PLEG): container finished" podID="c8a86b81-5e45-4c08-9d7c-9f1b8f11a201" containerID="a9169e3dacedcad9a34ece18477f80371b25604a1e192a888ba8676175e7e133" exitCode=0
Oct 31 01:41:26.864945 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:26.864874    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9" event={"ID":"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201","Type":"ContainerDied","Data":"a9169e3dacedcad9a34ece18477f80371b25604a1e192a888ba8676175e7e133"}
Oct 31 01:41:26.903780 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:26.903747    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:27.008191 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.008118    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:41:27.115447 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.115417    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-openshift-service-ca\") pod \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") "
Oct 31 01:41:27.115607 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.115455    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-maas-api-tls\") pod \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") "
Oct 31 01:41:27.115682 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.115659    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-kube-api-access-tdq54\" (UniqueName: \"kubernetes.io/projected/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-kube-api-access-tdq54\") pod \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") "
Oct 31 01:41:27.115787 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.115769    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-metrics-certs\") pod \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\" (UID: \"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201\") "
Oct 31 01:41:27.115867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.115806    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-openshift-service-ca" pod "c8a86b81-5e45-4c08-9d7c-9f1b8f11a201" (UID: "c8a86b81-5e45-4c08-9d7c-9f1b8f11a201"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:41:27.116290 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.116267    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:27.118071 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.118043    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-kube-api-access-tdq54" pod "c8a86b81-5e45-4c08-9d7c-9f1b8f11a201" (UID: "c8a86b81-5e45-4c08-9d7c-9f1b8f11a201"). InnerVolumeSpecName "kube-api-access-tdq54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:41:27.118211 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.118066    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-maas-api-tls" pod "c8a86b81-5e45-4c08-9d7c-9f1b8f11a201" (UID: "c8a86b81-5e45-4c08-9d7c-9f1b8f11a201"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:41:27.118382 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.118362    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-metrics-certs" pod "c8a86b81-5e45-4c08-9d7c-9f1b8f11a201" (UID: "c8a86b81-5e45-4c08-9d7c-9f1b8f11a201"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:41:27.217812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.217776    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:27.217812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.217805    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:27.217812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.217815    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdq54\" (UniqueName: \"kubernetes.io/projected/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201-kube-api-access-tdq54\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:27.389386 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.389355    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:27.641324 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.641250    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:27.871624 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.871591    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9" event={"ID":"c8a86b81-5e45-4c08-9d7c-9f1b8f11a201","Type":"ContainerDied","Data":"38868c994398e6c1e7f6d3467ce723e789e803a33d1e4e1a219b957e3bd582ff"}
Oct 31 01:41:27.871624 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.871623    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"
Oct 31 01:41:27.872126 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:27.871650    4476 scope.go:122] "RemoveContainer" containerID="a9169e3dacedcad9a34ece18477f80371b25604a1e192a888ba8676175e7e133"
Oct 31 01:41:28.038590 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:28.038559    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"]
Oct 31 01:41:28.087505 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:28.087477    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c9da8f-5f8678f8b9-vkdx9"]
Oct 31 01:41:28.454720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:28.454649    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a86b81-5e45-4c08-9d7c-9f1b8f11a201" path="/var/lib/kubelet/pods/c8a86b81-5e45-4c08-9d7c-9f1b8f11a201/volumes"
Oct 31 01:41:28.470251 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:28.470226    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:28.673141 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:28.673106    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:28.866071 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:28.866035    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:29.181251 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:41:29.181143    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:41:29.444199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:29.444098    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:30.779129 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:30.779101    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:41:30.804598 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:30.804577    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:41:30.863121 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:30.863099    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:41:33.616010 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:33.615978    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:34.343568 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:34.343537    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"]
Oct 31 01:41:34.464139 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:34.464102    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:36.805398 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:36.805364    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:37.112330 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:37.112245    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:38.208222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:38.208187    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:38.401162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:38.401127    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:38.592979 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:38.592952    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:39.899596 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:39.899565    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:43.231048 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:43.231017    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:49.364073 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.364020    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj" podUID="6d153817-c799-4114-8018-4ee979d4bb03" containerName="main" containerID="cri-o://cc32441e943b9a463c32527fdc41419922b5306e7ee7d6fb36eaffb7191557b9" gracePeriod=45
Oct 31 01:41:49.622876 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.622819    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:41:49.667632 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.667602    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:49.755127 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.755096    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d153817-c799-4114-8018-4ee979d4bb03-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d153817-c799-4114-8018-4ee979d4bb03-tls-certs\") pod \"6d153817-c799-4114-8018-4ee979d4bb03\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") "
Oct 31 01:41:49.755319 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.755222    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-dshm\") pod \"6d153817-c799-4114-8018-4ee979d4bb03\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") "
Oct 31 01:41:49.755319 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.755301    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-tmp-dir\") pod \"6d153817-c799-4114-8018-4ee979d4bb03\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") "
Oct 31 01:41:49.755437 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.755386    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d153817-c799-4114-8018-4ee979d4bb03-kube-api-access-cvwvl\" (UniqueName: \"kubernetes.io/projected/6d153817-c799-4114-8018-4ee979d4bb03-kube-api-access-cvwvl\") pod \"6d153817-c799-4114-8018-4ee979d4bb03\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") "
Oct 31 01:41:49.755437 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.755420    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-home\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-home\") pod \"6d153817-c799-4114-8018-4ee979d4bb03\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") "
Oct 31 01:41:49.755552 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.755473    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-model-cache\") pod \"6d153817-c799-4114-8018-4ee979d4bb03\" (UID: \"6d153817-c799-4114-8018-4ee979d4bb03\") "
Oct 31 01:41:49.755629 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.755603    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-tmp-dir" pod "6d153817-c799-4114-8018-4ee979d4bb03" (UID: "6d153817-c799-4114-8018-4ee979d4bb03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:41:49.756150 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.755833    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-home" pod "6d153817-c799-4114-8018-4ee979d4bb03" (UID: "6d153817-c799-4114-8018-4ee979d4bb03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:41:49.756305 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.756203    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-model-cache" pod "6d153817-c799-4114-8018-4ee979d4bb03" (UID: "6d153817-c799-4114-8018-4ee979d4bb03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:41:49.756499 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.756439    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:49.756499 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.756474    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:49.757974 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.757946    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d153817-c799-4114-8018-4ee979d4bb03-kube-api-access-cvwvl" pod "6d153817-c799-4114-8018-4ee979d4bb03" (UID: "6d153817-c799-4114-8018-4ee979d4bb03"). InnerVolumeSpecName "kube-api-access-cvwvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:41:49.758084 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.758047    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-dshm" pod "6d153817-c799-4114-8018-4ee979d4bb03" (UID: "6d153817-c799-4114-8018-4ee979d4bb03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:41:49.758151 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.758108    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d153817-c799-4114-8018-4ee979d4bb03-tls-certs" pod "6d153817-c799-4114-8018-4ee979d4bb03" (UID: "6d153817-c799-4114-8018-4ee979d4bb03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:41:49.857859 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.857831    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:49.857859 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.857858    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvwvl\" (UniqueName: \"kubernetes.io/projected/6d153817-c799-4114-8018-4ee979d4bb03-kube-api-access-cvwvl\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:49.858058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.857873    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d153817-c799-4114-8018-4ee979d4bb03-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:49.858058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:49.857886    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d153817-c799-4114-8018-4ee979d4bb03-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:50.001431 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:50.001354    4476 generic.go:356] "Generic (PLEG): container finished" podID="6d153817-c799-4114-8018-4ee979d4bb03" containerID="cc32441e943b9a463c32527fdc41419922b5306e7ee7d6fb36eaffb7191557b9" exitCode=0
Oct 31 01:41:50.001431 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:50.001402    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"
Oct 31 01:41:50.001431 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:50.001412    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj" event={"ID":"6d153817-c799-4114-8018-4ee979d4bb03","Type":"ContainerDied","Data":"cc32441e943b9a463c32527fdc41419922b5306e7ee7d6fb36eaffb7191557b9"}
Oct 31 01:41:50.001672 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:50.001449    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj" event={"ID":"6d153817-c799-4114-8018-4ee979d4bb03","Type":"ContainerDied","Data":"4f736e8365e1f8ebcdd2f1159d3591f51a78d73477d3541d162ca2c43f089a26"}
Oct 31 01:41:50.001672 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:50.001470    4476 scope.go:122] "RemoveContainer" containerID="cc32441e943b9a463c32527fdc41419922b5306e7ee7d6fb36eaffb7191557b9"
Oct 31 01:41:50.011270 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:50.011249    4476 scope.go:122] "RemoveContainer" containerID="cc32441e943b9a463c32527fdc41419922b5306e7ee7d6fb36eaffb7191557b9"
Oct 31 01:41:50.011527 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:41:50.011502    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc32441e943b9a463c32527fdc41419922b5306e7ee7d6fb36eaffb7191557b9\": container with ID starting with cc32441e943b9a463c32527fdc41419922b5306e7ee7d6fb36eaffb7191557b9 not found: ID does not exist" containerID="cc32441e943b9a463c32527fdc41419922b5306e7ee7d6fb36eaffb7191557b9"
Oct 31 01:41:50.011586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:50.011528    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc32441e943b9a463c32527fdc41419922b5306e7ee7d6fb36eaffb7191557b9"} err="failed to get container status \"cc32441e943b9a463c32527fdc41419922b5306e7ee7d6fb36eaffb7191557b9\": rpc error: code = NotFound desc = could not find container \"cc32441e943b9a463c32527fdc41419922b5306e7ee7d6fb36eaffb7191557b9\": container with ID starting with cc32441e943b9a463c32527fdc41419922b5306e7ee7d6fb36eaffb7191557b9 not found: ID does not exist"
Oct 31 01:41:50.164210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:50.164165    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"]
Oct 31 01:41:50.167538 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:50.167511    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-c9da8f/ipp-route-c9da8f-kserve-7b587cbc89-9x8hj"]
Oct 31 01:41:50.461630 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:50.461577    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d153817-c799-4114-8018-4ee979d4bb03" path="/var/lib/kubelet/pods/6d153817-c799-4114-8018-4ee979d4bb03/volumes"
Oct 31 01:41:53.403515 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:53.403474    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:54.417657 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:54.417620    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:54.605671 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:54.605640    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:59.184840 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:41:59.184795    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:41:59.221232 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:41:59.221204    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:00.413956 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:00.413923    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:00.612490 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:00.612452    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:00.793813 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:00.793779    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:04.213046 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:04.212956    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:09.527409 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:09.527370    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:10.011948 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:10.011914    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:10.674074 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:10.674039    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:14.676027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:14.675996    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:19.687013 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:19.686959    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:20.819518 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:20.819469    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:21.893306 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:21.893271    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:23.005377 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:23.005344    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:23.240377 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:23.240345    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:24.691495 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:24.691463    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:25.123151 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:25.123122    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:29.180034 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:42:29.179999    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:42:29.620575 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:29.620543    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:29.954580 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:29.954492    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:30.130557 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:30.130523    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:33.194909 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:33.194872    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:34.639466 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:34.639432    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:37.174398 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:37.174362    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:37.367896 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:37.367863    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:37.552866 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:37.552836    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:39.132476 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:39.132440    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"]
Oct 31 01:42:39.441917 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:39.441840    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:39.624322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:39.624289    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:39.813606 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:39.813559    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:40.364893 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:40.364858    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:42.879292 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:42.879262    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:43.068304 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.068268    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:43.248253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.248184    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"]
Oct 31 01:42:43.248664 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.248653    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c8a86b81-5e45-4c08-9d7c-9f1b8f11a201" containerName="maas-api"
Oct 31 01:42:43.248718 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.248666    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a86b81-5e45-4c08-9d7c-9f1b8f11a201" containerName="maas-api"
Oct 31 01:42:43.248718 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.248687    4476 cpu_manager.go:414] "containerMap: removing container" podUID="6d153817-c799-4114-8018-4ee979d4bb03" containerName="main"
Oct 31 01:42:43.248718 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.248692    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d153817-c799-4114-8018-4ee979d4bb03" containerName="main"
Oct 31 01:42:43.248820 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.248762    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d153817-c799-4114-8018-4ee979d4bb03" containerName="main"
Oct 31 01:42:43.248820 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.248775    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a86b81-5e45-4c08-9d7c-9f1b8f11a201" containerName="maas-api"
Oct 31 01:42:43.253056 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.253040    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"
Oct 31 01:42:43.323257 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.323202    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3defef0-a205-49c8-9bb1-ed3b194b329b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb\" (UID: \"a3defef0-a205-49c8-9bb1-ed3b194b329b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"
Oct 31 01:42:43.323521 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.323320    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpsbj\" (UniqueName: \"kubernetes.io/projected/a3defef0-a205-49c8-9bb1-ed3b194b329b-kube-api-access-hpsbj\") pod \"payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb\" (UID: \"a3defef0-a205-49c8-9bb1-ed3b194b329b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"
Oct 31 01:42:43.354530 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.354494    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ade881-dockercfg-2dnrq\""
Oct 31 01:42:43.354530 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.354493    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ade881\""
Oct 31 01:42:43.357235 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.357214    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"]
Oct 31 01:42:43.424855 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.424819    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3defef0-a205-49c8-9bb1-ed3b194b329b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb\" (UID: \"a3defef0-a205-49c8-9bb1-ed3b194b329b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"
Oct 31 01:42:43.425035 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.424910    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpsbj\" (UniqueName: \"kubernetes.io/projected/a3defef0-a205-49c8-9bb1-ed3b194b329b-kube-api-access-hpsbj\") pod \"payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb\" (UID: \"a3defef0-a205-49c8-9bb1-ed3b194b329b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"
Oct 31 01:42:43.425591 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.425555    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3defef0-a205-49c8-9bb1-ed3b194b329b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb\" (UID: \"a3defef0-a205-49c8-9bb1-ed3b194b329b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"
Oct 31 01:42:43.482086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.482055    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpsbj\" (UniqueName: \"kubernetes.io/projected/a3defef0-a205-49c8-9bb1-ed3b194b329b-kube-api-access-hpsbj\") pod \"payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb\" (UID: \"a3defef0-a205-49c8-9bb1-ed3b194b329b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"
Oct 31 01:42:43.566130 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.566105    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"
Oct 31 01:42:43.868016 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.867918    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"]
Oct 31 01:42:43.868267 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.868240    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2" podUID="bcf8b55a-0b39-4e7e-9603-70892f505536" containerName="maas-api" containerID="cri-o://e6b8165c46c8aa8a5948ecfc9352aec6a7045772fb7f8ed08565bf1badfc5baa" gracePeriod=30
Oct 31 01:42:43.983353 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:43.980417    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"]
Oct 31 01:42:44.015943 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.015912    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:44.120291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.120235    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:42:44.133226 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.133198    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8" podUID="98572193-ebb8-47e8-adc7-fbff21bc4c84" containerName="payload-pre-processing" containerID="cri-o://7dfb1ee079cdd80969d8a3f8944fadbf378bbf30ae25d596f7c1413808a1bd96" gracePeriod=25
Oct 31 01:42:44.234672 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.234165    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf8b55a-0b39-4e7e-9603-70892f505536-kube-api-access-d4hsn\" (UniqueName: \"kubernetes.io/projected/bcf8b55a-0b39-4e7e-9603-70892f505536-kube-api-access-d4hsn\") pod \"bcf8b55a-0b39-4e7e-9603-70892f505536\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") "
Oct 31 01:42:44.234672 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.234267    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf8b55a-0b39-4e7e-9603-70892f505536-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf8b55a-0b39-4e7e-9603-70892f505536-openshift-service-ca\") pod \"bcf8b55a-0b39-4e7e-9603-70892f505536\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") "
Oct 31 01:42:44.234672 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.234403    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-metrics-certs\") pod \"bcf8b55a-0b39-4e7e-9603-70892f505536\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") "
Oct 31 01:42:44.234672 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.234471    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-maas-api-tls\") pod \"bcf8b55a-0b39-4e7e-9603-70892f505536\" (UID: \"bcf8b55a-0b39-4e7e-9603-70892f505536\") "
Oct 31 01:42:44.235017 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.234883    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf8b55a-0b39-4e7e-9603-70892f505536-openshift-service-ca" pod "bcf8b55a-0b39-4e7e-9603-70892f505536" (UID: "bcf8b55a-0b39-4e7e-9603-70892f505536"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:44.239628 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.239599    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-metrics-certs" pod "bcf8b55a-0b39-4e7e-9603-70892f505536" (UID: "bcf8b55a-0b39-4e7e-9603-70892f505536"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:44.239825 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.239751    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf8b55a-0b39-4e7e-9603-70892f505536-kube-api-access-d4hsn" pod "bcf8b55a-0b39-4e7e-9603-70892f505536" (UID: "bcf8b55a-0b39-4e7e-9603-70892f505536"). InnerVolumeSpecName "kube-api-access-d4hsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:44.240636 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.240440    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-maas-api-tls" pod "bcf8b55a-0b39-4e7e-9603-70892f505536" (UID: "bcf8b55a-0b39-4e7e-9603-70892f505536"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:44.299539 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.299502    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb" event={"ID":"a3defef0-a205-49c8-9bb1-ed3b194b329b","Type":"ContainerStarted","Data":"1f89f0cf0c160d5ad444c1ca8d0f6e0aaef735cdb8de636fef9bd17ed141ce03"}
Oct 31 01:42:44.299843 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.299818    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb" event={"ID":"a3defef0-a205-49c8-9bb1-ed3b194b329b","Type":"ContainerStarted","Data":"6780788d327e9a94f0631d81d6a8c7b822375d622885c432ffbb341a2eabb114"}
Oct 31 01:42:44.299988 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.299975    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"
Oct 31 01:42:44.301449 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.301419    4476 generic.go:356] "Generic (PLEG): container finished" podID="bcf8b55a-0b39-4e7e-9603-70892f505536" containerID="e6b8165c46c8aa8a5948ecfc9352aec6a7045772fb7f8ed08565bf1badfc5baa" exitCode=0
Oct 31 01:42:44.301799 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.301498    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2" event={"ID":"bcf8b55a-0b39-4e7e-9603-70892f505536","Type":"ContainerDied","Data":"e6b8165c46c8aa8a5948ecfc9352aec6a7045772fb7f8ed08565bf1badfc5baa"}
Oct 31 01:42:44.301916 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.301835    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2" event={"ID":"bcf8b55a-0b39-4e7e-9603-70892f505536","Type":"ContainerDied","Data":"33ef4c5bf64a3f3a3762dae7a77a91160cd30edc4f075859294d33697100f50b"}
Oct 31 01:42:44.301916 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.301874    4476 scope.go:122] "RemoveContainer" containerID="e6b8165c46c8aa8a5948ecfc9352aec6a7045772fb7f8ed08565bf1badfc5baa"
Oct 31 01:42:44.301916 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.301885    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"
Oct 31 01:42:44.307510 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.307489    4476 generic.go:356] "Generic (PLEG): container finished" podID="98572193-ebb8-47e8-adc7-fbff21bc4c84" containerID="7dfb1ee079cdd80969d8a3f8944fadbf378bbf30ae25d596f7c1413808a1bd96" exitCode=0
Oct 31 01:42:44.307595 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.307544    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8" event={"ID":"98572193-ebb8-47e8-adc7-fbff21bc4c84","Type":"ContainerDied","Data":"7dfb1ee079cdd80969d8a3f8944fadbf378bbf30ae25d596f7c1413808a1bd96"}
Oct 31 01:42:44.317488 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.316817    4476 scope.go:122] "RemoveContainer" containerID="e6b8165c46c8aa8a5948ecfc9352aec6a7045772fb7f8ed08565bf1badfc5baa"
Oct 31 01:42:44.317488 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:42:44.317074    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b8165c46c8aa8a5948ecfc9352aec6a7045772fb7f8ed08565bf1badfc5baa\": container with ID starting with e6b8165c46c8aa8a5948ecfc9352aec6a7045772fb7f8ed08565bf1badfc5baa not found: ID does not exist" containerID="e6b8165c46c8aa8a5948ecfc9352aec6a7045772fb7f8ed08565bf1badfc5baa"
Oct 31 01:42:44.317488 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.317107    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b8165c46c8aa8a5948ecfc9352aec6a7045772fb7f8ed08565bf1badfc5baa"} err="failed to get container status \"e6b8165c46c8aa8a5948ecfc9352aec6a7045772fb7f8ed08565bf1badfc5baa\": rpc error: code = NotFound desc = could not find container \"e6b8165c46c8aa8a5948ecfc9352aec6a7045772fb7f8ed08565bf1badfc5baa\": container with ID starting with e6b8165c46c8aa8a5948ecfc9352aec6a7045772fb7f8ed08565bf1badfc5baa not found: ID does not exist"
Oct 31 01:42:44.336099 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.336074    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:44.336099 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.336099    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf8b55a-0b39-4e7e-9603-70892f505536-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:44.336282 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.336112    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4hsn\" (UniqueName: \"kubernetes.io/projected/bcf8b55a-0b39-4e7e-9603-70892f505536-kube-api-access-d4hsn\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:44.336282 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.336124    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf8b55a-0b39-4e7e-9603-70892f505536-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:44.385999 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.385975    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"
Oct 31 01:42:44.417399 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.417012    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb" podStartSLOduration=1.416995387 podStartE2EDuration="1.416995387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:44.415937325 +0000 UTC m=+2774.105820451" watchObservedRunningTime="2026-10-31 01:42:44.416995387 +0000 UTC m=+2774.106878515"
Oct 31 01:42:44.446035 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.445998    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:44.538466 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.538437    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98572193-ebb8-47e8-adc7-fbff21bc4c84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98572193-ebb8-47e8-adc7-fbff21bc4c84-plugins-config-volume\") pod \"98572193-ebb8-47e8-adc7-fbff21bc4c84\" (UID: \"98572193-ebb8-47e8-adc7-fbff21bc4c84\") "
Oct 31 01:42:44.538649 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.538633    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98572193-ebb8-47e8-adc7-fbff21bc4c84-kube-api-access-x2kgt\" (UniqueName: \"kubernetes.io/projected/98572193-ebb8-47e8-adc7-fbff21bc4c84-kube-api-access-x2kgt\") pod \"98572193-ebb8-47e8-adc7-fbff21bc4c84\" (UID: \"98572193-ebb8-47e8-adc7-fbff21bc4c84\") "
Oct 31 01:42:44.538809 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.538788    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98572193-ebb8-47e8-adc7-fbff21bc4c84-plugins-config-volume" pod "98572193-ebb8-47e8-adc7-fbff21bc4c84" (UID: "98572193-ebb8-47e8-adc7-fbff21bc4c84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:44.539030 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.539013    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98572193-ebb8-47e8-adc7-fbff21bc4c84-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:44.540833 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.540813    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98572193-ebb8-47e8-adc7-fbff21bc4c84-kube-api-access-x2kgt" pod "98572193-ebb8-47e8-adc7-fbff21bc4c84" (UID: "98572193-ebb8-47e8-adc7-fbff21bc4c84"). InnerVolumeSpecName "kube-api-access-x2kgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:44.622659 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.622634    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:44.639618 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.639551    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"]
Oct 31 01:42:44.639967 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.639947    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2kgt\" (UniqueName: \"kubernetes.io/projected/98572193-ebb8-47e8-adc7-fbff21bc4c84-kube-api-access-x2kgt\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:44.641719 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.641699    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a31828-85549b495b-k72q2"]
Oct 31 01:42:44.954498 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.954423    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"]
Oct 31 01:42:44.955058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.955036    4476 cpu_manager.go:414] "containerMap: removing container" podUID="bcf8b55a-0b39-4e7e-9603-70892f505536" containerName="maas-api"
Oct 31 01:42:44.955058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.955053    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf8b55a-0b39-4e7e-9603-70892f505536" containerName="maas-api"
Oct 31 01:42:44.955143 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.955081    4476 cpu_manager.go:414] "containerMap: removing container" podUID="98572193-ebb8-47e8-adc7-fbff21bc4c84" containerName="payload-pre-processing"
Oct 31 01:42:44.955143 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.955086    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98572193-ebb8-47e8-adc7-fbff21bc4c84" containerName="payload-pre-processing"
Oct 31 01:42:44.955222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.955156    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf8b55a-0b39-4e7e-9603-70892f505536" containerName="maas-api"
Oct 31 01:42:44.955222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.955168    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="98572193-ebb8-47e8-adc7-fbff21bc4c84" containerName="payload-pre-processing"
Oct 31 01:42:44.958492 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:44.958477    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.016227 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.010697    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-001a26-openshift-default-dockercfg-xw728\""
Oct 31 01:42:45.025525 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.025229    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"]
Oct 31 01:42:45.046765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.045110    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-workload-socket\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.046765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.045226    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-token\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.046765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.045260    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-credential-socket\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.046765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.045327    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-podinfo\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.046765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.045453    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9zb\" (UniqueName: \"kubernetes.io/projected/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-kube-api-access-4t9zb\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.046765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.045485    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istiod-ca-cert\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.046765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.045528    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-workload-certs\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.046765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.045618    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-envoy\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.046765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.045862    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-data\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.107417 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.107384    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:45.146757 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.146726    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9zb\" (UniqueName: \"kubernetes.io/projected/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-kube-api-access-4t9zb\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.146932 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.146760    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istiod-ca-cert\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.146932 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.146783    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-workload-certs\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.146932 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.146850    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-envoy\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.147049 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.146933    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-data\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.147049 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.146982    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-workload-socket\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.147151 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.147051    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-token\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.147151 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.147084    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-credential-socket\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.147151 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.147117    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-podinfo\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.147151 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.147119    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-workload-certs\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.147388 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.147305    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-data\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.147551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.147524    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istiod-ca-cert\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.147551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.147544    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-workload-socket\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.147677 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.147619    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-credential-socket\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.149657 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.149634    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-envoy\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.150325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.150303    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-podinfo\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.206480 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.206423    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-token\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.206735 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.206711    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9zb\" (UniqueName: \"kubernetes.io/projected/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-kube-api-access-4t9zb\") pod \"e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") " pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.275664 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.275636    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:45.315626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.315599    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"
Oct 31 01:42:45.315797 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.315643    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8" event={"ID":"98572193-ebb8-47e8-adc7-fbff21bc4c84","Type":"ContainerDied","Data":"d5837446cd7c99aba9b58597175722e823b5cb698c3736a72a4599e1774a2a4e"}
Oct 31 01:42:45.315797 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.315698    4476 scope.go:122] "RemoveContainer" containerID="7dfb1ee079cdd80969d8a3f8944fadbf378bbf30ae25d596f7c1413808a1bd96"
Oct 31 01:42:45.367271 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.367241    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:45.478377 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.478303    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"]
Oct 31 01:42:45.482070 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.482042    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6a9710-865f76ddb7-5p9l8"]
Oct 31 01:42:45.563748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.563718    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:45.700930 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.700908    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"]
Oct 31 01:42:45.703321 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:42:45.703297    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209964ea_0fcd_4a91_9f58_ad393cbc0b8a.slice/crio-c49628903482bd04bde644363fea36301344f5783ca1093e831d61302a4f623e WatchSource:0}: Error finding container c49628903482bd04bde644363fea36301344f5783ca1093e831d61302a4f623e: Status 404 returned error can't find the container with id c49628903482bd04bde644363fea36301344f5783ca1093e831d61302a4f623e
Oct 31 01:42:45.705765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.705735    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:42:45.705885 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.705798    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:42:45.705885 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.705828    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:42:45.963139 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:45.963104    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:46.277147 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:46.277115    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:46.323169 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:46.323139    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr" event={"ID":"209964ea-0fcd-4a91-9f58-ad393cbc0b8a","Type":"ContainerStarted","Data":"71dd802f7093920c7be9c2374d4842ec1d51274ad0157918be83b59e4ce5c173"}
Oct 31 01:42:46.323349 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:46.323208    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr" event={"ID":"209964ea-0fcd-4a91-9f58-ad393cbc0b8a","Type":"ContainerStarted","Data":"c49628903482bd04bde644363fea36301344f5783ca1093e831d61302a4f623e"}
Oct 31 01:42:46.439402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:46.439346    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr" podStartSLOduration=2.439332905 podStartE2EDuration="2.439332905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:46.437853284 +0000 UTC m=+2776.127736421" watchObservedRunningTime="2026-10-31 01:42:46.439332905 +0000 UTC m=+2776.129216030"
Oct 31 01:42:46.453607 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:46.453559    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98572193-ebb8-47e8-adc7-fbff21bc4c84" path="/var/lib/kubelet/pods/98572193-ebb8-47e8-adc7-fbff21bc4c84/volumes"
Oct 31 01:42:46.454015 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:46.454001    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf8b55a-0b39-4e7e-9603-70892f505536" path="/var/lib/kubelet/pods/bcf8b55a-0b39-4e7e-9603-70892f505536/volumes"
Oct 31 01:42:47.065056 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:47.065031    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:47.273015 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:47.272983    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:47.275878 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:47.275857    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:47.281861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:47.281839    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:47.329315 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:47.329259    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:47.330016 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:47.329998    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:42:47.503870 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:47.503819    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:48.111307 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:48.111279    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:49.334470 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:49.334438    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:49.526396 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:49.526370    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:49.707669 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:49.707562    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:52.188573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:52.188526    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:52.505478 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:52.505405    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:52.689079 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:52.689043    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:54.392979 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:54.392952    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:54.586531 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:54.586493    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:54.783439 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:54.783401    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:55.318318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:55.318290    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"
Oct 31 01:42:57.013878 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:57.013845    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:57.511225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:42:57.511165    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:59.180708 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:42:59.180676    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:43:00.699595 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:00.699565    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:01.343680 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:01.343640    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:01.542308 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:01.542276    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:02.411729 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:02.411696    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:03.119627 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:03.119593    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:03.304541 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:03.304514    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:03.427494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:03.427410    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"]
Oct 31 01:43:03.427898 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:03.427621    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9" podUID="a535f499-71aa-4da2-823a-16d687bd5d92" containerName="istio-proxy" containerID="cri-o://27ed3c911be4746de61750ad721f406c771a6147ef6a513594965108168a8263" gracePeriod=30
Oct 31 01:43:03.510027 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:03.509992    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:05.636783 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:05.636750    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:05.828558 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:05.828517    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:06.868449 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:06.868422    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:07.417811 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:07.417782    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:07.595674 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:07.595646    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:08.683891 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.683866    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:43:08.787473 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.787424    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:08.802748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.802721    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-workload-certs\") pod \"a535f499-71aa-4da2-823a-16d687bd5d92\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") "
Oct 31 01:43:08.802898 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.802826    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-istio-envoy\") pod \"a535f499-71aa-4da2-823a-16d687bd5d92\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") "
Oct 31 01:43:08.802898 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.802868    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a535f499-71aa-4da2-823a-16d687bd5d92-istio-token\" (UniqueName: \"kubernetes.io/projected/a535f499-71aa-4da2-823a-16d687bd5d92-istio-token\") pod \"a535f499-71aa-4da2-823a-16d687bd5d92\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") "
Oct 31 01:43:08.802898 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.802886    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-credential-socket\") pod \"a535f499-71aa-4da2-823a-16d687bd5d92\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") "
Oct 31 01:43:08.803043 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.802954    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a535f499-71aa-4da2-823a-16d687bd5d92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a535f499-71aa-4da2-823a-16d687bd5d92-istiod-ca-cert\") pod \"a535f499-71aa-4da2-823a-16d687bd5d92\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") "
Oct 31 01:43:08.803043 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.803002    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-workload-certs" pod "a535f499-71aa-4da2-823a-16d687bd5d92" (UID: "a535f499-71aa-4da2-823a-16d687bd5d92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:08.803150 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.803080    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-istio-data\") pod \"a535f499-71aa-4da2-823a-16d687bd5d92\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") "
Oct 31 01:43:08.803150 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.803121    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a535f499-71aa-4da2-823a-16d687bd5d92-kube-api-access-w95v6\" (UniqueName: \"kubernetes.io/projected/a535f499-71aa-4da2-823a-16d687bd5d92-kube-api-access-w95v6\") pod \"a535f499-71aa-4da2-823a-16d687bd5d92\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") "
Oct 31 01:43:08.803275 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.803194    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a535f499-71aa-4da2-823a-16d687bd5d92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a535f499-71aa-4da2-823a-16d687bd5d92-istio-podinfo\") pod \"a535f499-71aa-4da2-823a-16d687bd5d92\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") "
Oct 31 01:43:08.803336 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.803274    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-workload-socket\") pod \"a535f499-71aa-4da2-823a-16d687bd5d92\" (UID: \"a535f499-71aa-4da2-823a-16d687bd5d92\") "
Oct 31 01:43:08.803543 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.803492    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a535f499-71aa-4da2-823a-16d687bd5d92-istiod-ca-cert" pod "a535f499-71aa-4da2-823a-16d687bd5d92" (UID: "a535f499-71aa-4da2-823a-16d687bd5d92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:08.803646 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.803599    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-istio-data" pod "a535f499-71aa-4da2-823a-16d687bd5d92" (UID: "a535f499-71aa-4da2-823a-16d687bd5d92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:08.803732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.803678    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-workload-socket" pod "a535f499-71aa-4da2-823a-16d687bd5d92" (UID: "a535f499-71aa-4da2-823a-16d687bd5d92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:08.803847 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.803828    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a535f499-71aa-4da2-823a-16d687bd5d92-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:08.803968 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.803850    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:08.803968 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.803865    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:08.803968 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.803876    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:08.804120 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.804065    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-credential-socket" pod "a535f499-71aa-4da2-823a-16d687bd5d92" (UID: "a535f499-71aa-4da2-823a-16d687bd5d92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:08.805565 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.805529    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-istio-envoy" pod "a535f499-71aa-4da2-823a-16d687bd5d92" (UID: "a535f499-71aa-4da2-823a-16d687bd5d92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:08.806137 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.806111    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a535f499-71aa-4da2-823a-16d687bd5d92-istio-podinfo" pod "a535f499-71aa-4da2-823a-16d687bd5d92" (UID: "a535f499-71aa-4da2-823a-16d687bd5d92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:43:08.806274 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.806234    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a535f499-71aa-4da2-823a-16d687bd5d92-istio-token" pod "a535f499-71aa-4da2-823a-16d687bd5d92" (UID: "a535f499-71aa-4da2-823a-16d687bd5d92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:08.806274 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.806254    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a535f499-71aa-4da2-823a-16d687bd5d92-kube-api-access-w95v6" pod "a535f499-71aa-4da2-823a-16d687bd5d92" (UID: "a535f499-71aa-4da2-823a-16d687bd5d92"). InnerVolumeSpecName "kube-api-access-w95v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:08.905234 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.905203    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a535f499-71aa-4da2-823a-16d687bd5d92-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:08.905234 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.905234    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:08.905407 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.905243    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a535f499-71aa-4da2-823a-16d687bd5d92-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:08.905407 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.905252    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a535f499-71aa-4da2-823a-16d687bd5d92-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:08.905407 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:08.905261    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w95v6\" (UniqueName: \"kubernetes.io/projected/a535f499-71aa-4da2-823a-16d687bd5d92-kube-api-access-w95v6\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:09.469557 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:09.469510    4476 generic.go:356] "Generic (PLEG): container finished" podID="a535f499-71aa-4da2-823a-16d687bd5d92" containerID="27ed3c911be4746de61750ad721f406c771a6147ef6a513594965108168a8263" exitCode=0
Oct 31 01:43:09.469779 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:09.469596    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"
Oct 31 01:43:09.469779 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:09.469707    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9" event={"ID":"a535f499-71aa-4da2-823a-16d687bd5d92","Type":"ContainerDied","Data":"27ed3c911be4746de61750ad721f406c771a6147ef6a513594965108168a8263"}
Oct 31 01:43:09.469910 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:09.469794    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9" event={"ID":"a535f499-71aa-4da2-823a-16d687bd5d92","Type":"ContainerDied","Data":"ace828b58647d04902c2cb8a627609ab84cacb8c55d6957f20bec9a71436ed45"}
Oct 31 01:43:09.469910 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:09.469825    4476 scope.go:122] "RemoveContainer" containerID="27ed3c911be4746de61750ad721f406c771a6147ef6a513594965108168a8263"
Oct 31 01:43:09.479939 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:09.479924    4476 scope.go:122] "RemoveContainer" containerID="c4d60dffbc9a377a159ec1828d47e4fa900d14243a1c58a247506fd48361c480"
Oct 31 01:43:09.488331 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:09.488315    4476 scope.go:122] "RemoveContainer" containerID="27ed3c911be4746de61750ad721f406c771a6147ef6a513594965108168a8263"
Oct 31 01:43:09.488583 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:43:09.488565    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ed3c911be4746de61750ad721f406c771a6147ef6a513594965108168a8263\": container with ID starting with 27ed3c911be4746de61750ad721f406c771a6147ef6a513594965108168a8263 not found: ID does not exist" containerID="27ed3c911be4746de61750ad721f406c771a6147ef6a513594965108168a8263"
Oct 31 01:43:09.488632 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:09.488590    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ed3c911be4746de61750ad721f406c771a6147ef6a513594965108168a8263"} err="failed to get container status \"27ed3c911be4746de61750ad721f406c771a6147ef6a513594965108168a8263\": rpc error: code = NotFound desc = could not find container \"27ed3c911be4746de61750ad721f406c771a6147ef6a513594965108168a8263\": container with ID starting with 27ed3c911be4746de61750ad721f406c771a6147ef6a513594965108168a8263 not found: ID does not exist"
Oct 31 01:43:09.488632 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:09.488607    4476 scope.go:122] "RemoveContainer" containerID="c4d60dffbc9a377a159ec1828d47e4fa900d14243a1c58a247506fd48361c480"
Oct 31 01:43:09.488826 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:43:09.488813    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d60dffbc9a377a159ec1828d47e4fa900d14243a1c58a247506fd48361c480\": container with ID starting with c4d60dffbc9a377a159ec1828d47e4fa900d14243a1c58a247506fd48361c480 not found: ID does not exist" containerID="c4d60dffbc9a377a159ec1828d47e4fa900d14243a1c58a247506fd48361c480"
Oct 31 01:43:09.488867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:09.488828    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d60dffbc9a377a159ec1828d47e4fa900d14243a1c58a247506fd48361c480"} err="failed to get container status \"c4d60dffbc9a377a159ec1828d47e4fa900d14243a1c58a247506fd48361c480\": rpc error: code = NotFound desc = could not find container \"c4d60dffbc9a377a159ec1828d47e4fa900d14243a1c58a247506fd48361c480\": container with ID starting with c4d60dffbc9a377a159ec1828d47e4fa900d14243a1c58a247506fd48361c480 not found: ID does not exist"
Oct 31 01:43:09.684335 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:09.684305    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"]
Oct 31 01:43:09.735096 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:09.735021    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a31828-openshift-default-675c75fbc5-tbrs9"]
Oct 31 01:43:10.454750 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:10.454665    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a535f499-71aa-4da2-823a-16d687bd5d92" path="/var/lib/kubelet/pods/a535f499-71aa-4da2-823a-16d687bd5d92/volumes"
Oct 31 01:43:10.643426 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:10.643395    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:10.888697 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:10.888669    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:13.781164 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:13.781117    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:14.139813 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:14.139743    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:14.414794 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:14.414715    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:15.075326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:15.075291    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:18.881726 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:18.881694    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:20.899726 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:20.899695    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:21.077063 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:21.077032    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:21.257955 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:21.257876    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:21.444359 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:21.444327    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:23.758114 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:23.758084    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:28.485199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:28.485138    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:28.673931 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:28.673898    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:29.180191 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:43:29.180141    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:43:33.415978 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:33.415945    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:38.207636 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:38.207603    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:42.948134 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:42.948035    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:43.169934 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:43.169884    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:43.372630 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:43.372595    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:45.273091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:45.273063    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:48.040291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:48.040263    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:48.966096 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:48.966066    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:50.743658 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:50.743629    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:52.831433 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:52.831401    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:57.715165 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:43:57.715121    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:59.181241 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:43:59.181202    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:44:02.478913 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:02.478886    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:02.689678 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:02.689646    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:02.903864 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:02.903826    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:07.277433 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.277402    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"]
Oct 31 01:44:07.277906 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.277891    4476 cpu_manager.go:414] "containerMap: removing container" podUID="a535f499-71aa-4da2-823a-16d687bd5d92" containerName="istio-proxy"
Oct 31 01:44:07.277949 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.277907    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a535f499-71aa-4da2-823a-16d687bd5d92" containerName="istio-proxy"
Oct 31 01:44:07.277949 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.277921    4476 cpu_manager.go:414] "containerMap: removing container" podUID="a535f499-71aa-4da2-823a-16d687bd5d92" containerName="istio-proxy"
Oct 31 01:44:07.277949 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.277927    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a535f499-71aa-4da2-823a-16d687bd5d92" containerName="istio-proxy"
Oct 31 01:44:07.278047 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.278016    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a535f499-71aa-4da2-823a-16d687bd5d92" containerName="istio-proxy"
Oct 31 01:44:07.278047 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.278027    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a535f499-71aa-4da2-823a-16d687bd5d92" containerName="istio-proxy"
Oct 31 01:44:07.281357 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.281340    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.336279 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.336258    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e621db-openshift-default-dockercfg-p8kgb\""
Oct 31 01:44:07.341783 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.341756    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrlsk\" (UniqueName: \"kubernetes.io/projected/50c47e81-c94e-4455-88d3-1d4b1c311d75-kube-api-access-qrlsk\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.341915 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.341792    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.341915 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.341826    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.342041 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.341945    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.342041 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.342023    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.342151 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.342052    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c47e81-c94e-4455-88d3-1d4b1c311d75-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.342151 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.342097    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.342283 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.342159    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.342283 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.342254    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.387186 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.387140    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"]
Oct 31 01:44:07.443620 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.443586    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.443620 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.443623    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.443813 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.443761    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.443813 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.443802    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c47e81-c94e-4455-88d3-1d4b1c311d75-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.443901 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.443849    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.443901 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.443895    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.444016 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.443944    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.444074 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.444015    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.444129 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.444081    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.444129 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.444099    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrlsk\" (UniqueName: \"kubernetes.io/projected/50c47e81-c94e-4455-88d3-1d4b1c311d75-kube-api-access-qrlsk\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.444283 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.444135    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.444370 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.444344    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.444436 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.444400    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.444675 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.444652    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c47e81-c94e-4455-88d3-1d4b1c311d75-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.446126 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.446104    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.446633 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.446610    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.504704 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.504678    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.504822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.504737    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrlsk\" (UniqueName: \"kubernetes.io/projected/50c47e81-c94e-4455-88d3-1d4b1c311d75-kube-api-access-qrlsk\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:07.572579 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.572547    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:07.594540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:07.594514    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:08.020016 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:08.019991    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"]
Oct 31 01:44:08.022069 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:44:08.022039    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c47e81_c94e_4455_88d3_1d4b1c311d75.slice/crio-e20730a1f63817544ace5c01b1d0b580ca15208681f2abf7b93e249f1481c960 WatchSource:0}: Error finding container e20730a1f63817544ace5c01b1d0b580ca15208681f2abf7b93e249f1481c960: Status 404 returned error can't find the container with id e20730a1f63817544ace5c01b1d0b580ca15208681f2abf7b93e249f1481c960
Oct 31 01:44:08.024582 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:08.024544    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:44:08.024699 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:08.024614    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:44:08.024699 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:08.024649    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:44:08.800091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:08.800051    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh" event={"ID":"50c47e81-c94e-4455-88d3-1d4b1c311d75","Type":"ContainerStarted","Data":"a396658eabc427dfa8e714d631a321fbb650ad5e406ffb5431b4553d20aedb75"}
Oct 31 01:44:08.800521 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:08.800113    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh" event={"ID":"50c47e81-c94e-4455-88d3-1d4b1c311d75","Type":"ContainerStarted","Data":"e20730a1f63817544ace5c01b1d0b580ca15208681f2abf7b93e249f1481c960"}
Oct 31 01:44:08.981384 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:08.981303    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh" podStartSLOduration=1.981280382 podStartE2EDuration="1.981280382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:08.979050032 +0000 UTC m=+2858.668933161" watchObservedRunningTime="2026-10-31 01:44:08.981280382 +0000 UTC m=+2858.671163510"
Oct 31 01:44:09.595367 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:09.595328    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:09.596925 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:09.596897    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 01:44:09.597082 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:09.596956    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh" podUID="50c47e81-c94e-4455-88d3-1d4b1c311d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 01:44:10.595501 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:10.595464    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 01:44:10.595925 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:10.595517    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh" podUID="50c47e81-c94e-4455-88d3-1d4b1c311d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 01:44:11.595365 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:11.595328    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 01:44:11.595565 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:11.595393    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh" podUID="50c47e81-c94e-4455-88d3-1d4b1c311d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 01:44:12.413677 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:12.413639    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:12.596160 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:12.596117    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 01:44:12.596838 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:12.596792    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh" podUID="50c47e81-c94e-4455-88d3-1d4b1c311d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 01:44:13.595697 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:13.595665    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 01:44:13.595899 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:13.595721    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh" podUID="50c47e81-c94e-4455-88d3-1d4b1c311d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 01:44:14.595864 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:14.595826    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 01:44:14.596296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:14.595895    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh" podUID="50c47e81-c94e-4455-88d3-1d4b1c311d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 01:44:16.637317 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:16.637275    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:16.637888 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:16.637350    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh" podUID="50c47e81-c94e-4455-88d3-1d4b1c311d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:17.225489 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:17.225453    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:17.225762 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:17.225743    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:17.226514 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:17.226496    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:44:17.441217 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:17.441153    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:21.538794 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:21.538762    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:22.202042 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:22.201967    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:24.908735 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:24.908703    4476 generic.go:356] "Generic (PLEG): container finished" podID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" containerID="71dd802f7093920c7be9c2374d4842ec1d51274ad0157918be83b59e4ce5c173" exitCode=0
Oct 31 01:44:24.909245 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:24.908788    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr" event={"ID":"209964ea-0fcd-4a91-9f58-ad393cbc0b8a","Type":"ContainerDied","Data":"71dd802f7093920c7be9c2374d4842ec1d51274ad0157918be83b59e4ce5c173"}
Oct 31 01:44:24.909245 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:24.909153    4476 scope.go:122] "RemoveContainer" containerID="71dd802f7093920c7be9c2374d4842ec1d51274ad0157918be83b59e4ce5c173"
Oct 31 01:44:24.910593 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:24.910564    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:44:24.910660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:24.910631    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:44:24.910696 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:24.910665    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:44:25.917168 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:25.917133    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr" event={"ID":"209964ea-0fcd-4a91-9f58-ad393cbc0b8a","Type":"ContainerStarted","Data":"b3ff90d7279d38c25ce3100dde332cb15d213fa786bda0d162a5fbc86eb4d7ee"}
Oct 31 01:44:26.284200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:26.279584    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:44:26.284200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:26.279717    4476 patch_prober.go:29] interesting pod/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:44:26.284200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:26.279761    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr" podUID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 31 01:44:27.016560 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:27.016530    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:27.284348 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:27.284313    4476 patch_prober.go:29] interesting pod/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:44:27.284543 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:27.284376    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr" podUID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 31 01:44:28.284540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:28.284501    4476 patch_prober.go:29] interesting pod/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:44:28.285095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:28.284586    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr" podUID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 31 01:44:29.203650 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:44:29.203607    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:44:29.284599 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:29.284560    4476 patch_prober.go:29] interesting pod/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:44:29.285000 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:29.284622    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr" podUID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 31 01:44:30.279202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:30.278594    4476 patch_prober.go:29] interesting pod/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:44:30.279202 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:30.278674    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr" podUID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 31 01:44:31.280763 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:31.280733    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:44:31.281275 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:31.281095    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:44:31.281793 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:31.281777    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:44:31.821381 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:31.821343    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:36.846787 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:36.846756    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:41.611942 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:41.611881    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:44.788868 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:44.788812    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:45.801908 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:45.801878    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:46.546919 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:46.546877    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:49.482434 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.482403    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"]
Oct 31 01:44:49.486441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.486424    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.540496 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.540460    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-25b9ff\"/\"kube-root-ca.crt\""
Oct 31 01:44:49.543807 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.543780    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"]
Oct 31 01:44:49.589861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.589834    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-25b9ff\"/\"openshift-service-ca.crt\""
Oct 31 01:44:49.590006 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.589933    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-25b9ff\"/\"default-dockercfg-pjxc5\""
Oct 31 01:44:49.590223 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.590207    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-25b9ff\"/\"praxis-route-25b9ff-kserve-self-signed-certs\""
Oct 31 01:44:49.619405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.619374    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:49.664315 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.664287    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rvjm\" (UniqueName: \"kubernetes.io/projected/da4721b0-ca4f-4538-829d-0590b02ccfdf-kube-api-access-4rvjm\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.664494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.664421    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4721b0-ca4f-4538-829d-0590b02ccfdf-tls-certs\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.664494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.664454    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-tmp-dir\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.664592 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.664546    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-dshm\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.664592 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.664569    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-model-cache\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.664592 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.664589    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-home\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.765148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.765069    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-home\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.765148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.765132    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rvjm\" (UniqueName: \"kubernetes.io/projected/da4721b0-ca4f-4538-829d-0590b02ccfdf-kube-api-access-4rvjm\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.765367 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.765312    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4721b0-ca4f-4538-829d-0590b02ccfdf-tls-certs\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.765367 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.765352    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-tmp-dir\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.765473 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.765461    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-dshm\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.765538 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.765484    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-model-cache\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.765748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.765726    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-tmp-dir\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.765830 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.765761    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-home\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.765830 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.765802    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-model-cache\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.767760 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.767742    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-dshm\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.768057 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.768039    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4721b0-ca4f-4538-829d-0590b02ccfdf-tls-certs\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:49.822442 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:49.822411    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rvjm\" (UniqueName: \"kubernetes.io/projected/da4721b0-ca4f-4538-829d-0590b02ccfdf-kube-api-access-4rvjm\") pod \"praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") " pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:50.100388 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:50.100362    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:50.472760 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:50.472734    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"]
Oct 31 01:44:50.474353 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:44:50.474327    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda4721b0_ca4f_4538_829d_0590b02ccfdf.slice/crio-6dad1f9098652450d93fa8a0b20b47110e35a6cd980a1aea78df7203027b908b WatchSource:0}: Error finding container 6dad1f9098652450d93fa8a0b20b47110e35a6cd980a1aea78df7203027b908b: Status 404 returned error can't find the container with id 6dad1f9098652450d93fa8a0b20b47110e35a6cd980a1aea78df7203027b908b
Oct 31 01:44:51.056639 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:51.056605    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p" event={"ID":"da4721b0-ca4f-4538-829d-0590b02ccfdf","Type":"ContainerStarted","Data":"4d25fe49d772658d7ac221c8cae5c0db430f6137192fdb4930784a0220dfdc60"}
Oct 31 01:44:51.057089 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:51.056649    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p" event={"ID":"da4721b0-ca4f-4538-829d-0590b02ccfdf","Type":"ContainerStarted","Data":"6dad1f9098652450d93fa8a0b20b47110e35a6cd980a1aea78df7203027b908b"}
Oct 31 01:44:51.100761 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:51.100735    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:44:51.173325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:51.173279    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p" podStartSLOduration=2.173265828 podStartE2EDuration="2.173265828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:51.170911427 +0000 UTC m=+2900.860794555" watchObservedRunningTime="2026-10-31 01:44:51.173265828 +0000 UTC m=+2900.863148955"
Oct 31 01:44:51.710239 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:51.710209    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:51.945769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:51.945738    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:56.612662 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:44:56.612624    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:59.183168 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:44:59.183124    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:45:00.100684 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.100652    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:45:00.113767 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.113735    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:45:00.125114 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.125089    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:45:00.228869 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.228838    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"]
Oct 31 01:45:00.232623 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.232603    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"
Oct 31 01:45:00.281376 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.281349    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbqkr\" (UniqueName: \"kubernetes.io/projected/28fbc476-4f37-403e-9bd0-dece71e21460-kube-api-access-pbqkr\") pod \"maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz\" (UID: \"28fbc476-4f37-403e-9bd0-dece71e21460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"
Oct 31 01:45:00.281526 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.281418    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fbc476-4f37-403e-9bd0-dece71e21460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz\" (UID: \"28fbc476-4f37-403e-9bd0-dece71e21460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"
Oct 31 01:45:00.347400 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.347363    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\""
Oct 31 01:45:00.383131 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.383039    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbqkr\" (UniqueName: \"kubernetes.io/projected/28fbc476-4f37-403e-9bd0-dece71e21460-kube-api-access-pbqkr\") pod \"maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz\" (UID: \"28fbc476-4f37-403e-9bd0-dece71e21460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"
Oct 31 01:45:00.383131 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.383108    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fbc476-4f37-403e-9bd0-dece71e21460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz\" (UID: \"28fbc476-4f37-403e-9bd0-dece71e21460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"
Oct 31 01:45:00.383693 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.383670    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fbc476-4f37-403e-9bd0-dece71e21460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz\" (UID: \"28fbc476-4f37-403e-9bd0-dece71e21460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"
Oct 31 01:45:00.399616 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.399585    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"]
Oct 31 01:45:00.449258 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.449228    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbqkr\" (UniqueName: \"kubernetes.io/projected/28fbc476-4f37-403e-9bd0-dece71e21460-kube-api-access-pbqkr\") pod \"maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz\" (UID: \"28fbc476-4f37-403e-9bd0-dece71e21460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"
Oct 31 01:45:00.547440 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.547409    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"
Oct 31 01:45:00.933618 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:00.933596    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"]
Oct 31 01:45:00.935880 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:45:00.935839    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fbc476_4f37_403e_9bd0_dece71e21460.slice/crio-8e09aaa19c6586458de6bae3d96c482597992f418a6521c1874f87d382fea9cd WatchSource:0}: Error finding container 8e09aaa19c6586458de6bae3d96c482597992f418a6521c1874f87d382fea9cd: Status 404 returned error can't find the container with id 8e09aaa19c6586458de6bae3d96c482597992f418a6521c1874f87d382fea9cd
Oct 31 01:45:01.043347 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:01.043313    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:01.114773 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:01.114740    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz" event={"ID":"28fbc476-4f37-403e-9bd0-dece71e21460","Type":"ContainerStarted","Data":"8e09aaa19c6586458de6bae3d96c482597992f418a6521c1874f87d382fea9cd"}
Oct 31 01:45:01.523820 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:01.523736    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:02.121836 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:02.121801    4476 generic.go:356] "Generic (PLEG): container finished" podID="28fbc476-4f37-403e-9bd0-dece71e21460" containerID="98eb0e230fcc74f4b5aea0fa46ca670b68cbe73ea2f0e524527e86a27b911ac3" exitCode=60
Oct 31 01:45:02.122018 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:02.121893    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz" event={"ID":"28fbc476-4f37-403e-9bd0-dece71e21460","Type":"ContainerDied","Data":"98eb0e230fcc74f4b5aea0fa46ca670b68cbe73ea2f0e524527e86a27b911ac3"}
Oct 31 01:45:02.122114 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:02.122101    4476 scope.go:122] "RemoveContainer" containerID="98eb0e230fcc74f4b5aea0fa46ca670b68cbe73ea2f0e524527e86a27b911ac3"
Oct 31 01:45:03.129585 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:03.129548    4476 generic.go:356] "Generic (PLEG): container finished" podID="28fbc476-4f37-403e-9bd0-dece71e21460" containerID="ace15153dc84bc74749e3d2e342481af2e11b5ecbee6053880240949ed017783" exitCode=60
Oct 31 01:45:03.129984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:03.129635    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz" event={"ID":"28fbc476-4f37-403e-9bd0-dece71e21460","Type":"ContainerDied","Data":"ace15153dc84bc74749e3d2e342481af2e11b5ecbee6053880240949ed017783"}
Oct 31 01:45:03.129984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:03.129700    4476 scope.go:122] "RemoveContainer" containerID="98eb0e230fcc74f4b5aea0fa46ca670b68cbe73ea2f0e524527e86a27b911ac3"
Oct 31 01:45:03.130103 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:03.129987    4476 scope.go:122] "RemoveContainer" containerID="ace15153dc84bc74749e3d2e342481af2e11b5ecbee6053880240949ed017783"
Oct 31 01:45:03.130275 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:45:03.130254    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz_odh-ai-gateway-infra(28fbc476-4f37-403e-9bd0-dece71e21460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz" podUID="28fbc476-4f37-403e-9bd0-dece71e21460"
Oct 31 01:45:04.137010 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:04.136970    4476 scope.go:122] "RemoveContainer" containerID="ace15153dc84bc74749e3d2e342481af2e11b5ecbee6053880240949ed017783"
Oct 31 01:45:04.137406 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:45:04.137224    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz_odh-ai-gateway-infra(28fbc476-4f37-403e-9bd0-dece71e21460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz" podUID="28fbc476-4f37-403e-9bd0-dece71e21460"
Oct 31 01:45:06.247121 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:06.247062    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:08.517457 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:08.517423    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:09.094042 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:09.094006    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:09.271830 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:09.271800    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:09.609255 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:09.609226    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:10.507227 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:10.507141    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"]
Oct 31 01:45:10.507374 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:10.507342    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld" podUID="89ee38a2-7a31-49ca-b903-af33df249cfa" containerName="maas-api" containerID="cri-o://b1a13934fb0676463765e397422ff8ec57ef71ecca0049b3e06201894810fdbb" gracePeriod=30
Oct 31 01:45:10.652157 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:10.652123    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:10.766806 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:10.766744    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:45:10.901883 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:10.901855    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89ee38a2-7a31-49ca-b903-af33df249cfa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ee38a2-7a31-49ca-b903-af33df249cfa-openshift-service-ca\") pod \"89ee38a2-7a31-49ca-b903-af33df249cfa\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") "
Oct 31 01:45:10.902298 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:10.901897    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-metrics-certs\") pod \"89ee38a2-7a31-49ca-b903-af33df249cfa\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") "
Oct 31 01:45:10.902298 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:10.901952    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-maas-api-tls\") pod \"89ee38a2-7a31-49ca-b903-af33df249cfa\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") "
Oct 31 01:45:10.902298 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:10.902097    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ee38a2-7a31-49ca-b903-af33df249cfa-kube-api-access-74hpl\" (UniqueName: \"kubernetes.io/projected/89ee38a2-7a31-49ca-b903-af33df249cfa-kube-api-access-74hpl\") pod \"89ee38a2-7a31-49ca-b903-af33df249cfa\" (UID: \"89ee38a2-7a31-49ca-b903-af33df249cfa\") "
Oct 31 01:45:10.902497 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:10.902342    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ee38a2-7a31-49ca-b903-af33df249cfa-openshift-service-ca" pod "89ee38a2-7a31-49ca-b903-af33df249cfa" (UID: "89ee38a2-7a31-49ca-b903-af33df249cfa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:10.902977 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:10.902881    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ee38a2-7a31-49ca-b903-af33df249cfa-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:10.904514 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:10.904474    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-maas-api-tls" pod "89ee38a2-7a31-49ca-b903-af33df249cfa" (UID: "89ee38a2-7a31-49ca-b903-af33df249cfa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:45:10.904634 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:10.904536    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-metrics-certs" pod "89ee38a2-7a31-49ca-b903-af33df249cfa" (UID: "89ee38a2-7a31-49ca-b903-af33df249cfa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:45:10.905098 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:10.905068    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ee38a2-7a31-49ca-b903-af33df249cfa-kube-api-access-74hpl" pod "89ee38a2-7a31-49ca-b903-af33df249cfa" (UID: "89ee38a2-7a31-49ca-b903-af33df249cfa"). InnerVolumeSpecName "kube-api-access-74hpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:11.004311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.004282    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:11.004311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.004306    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74hpl\" (UniqueName: \"kubernetes.io/projected/89ee38a2-7a31-49ca-b903-af33df249cfa-kube-api-access-74hpl\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:11.004311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.004317    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ee38a2-7a31-49ca-b903-af33df249cfa-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:11.177014 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.176973    4476 generic.go:356] "Generic (PLEG): container finished" podID="89ee38a2-7a31-49ca-b903-af33df249cfa" containerID="b1a13934fb0676463765e397422ff8ec57ef71ecca0049b3e06201894810fdbb" exitCode=0
Oct 31 01:45:11.177224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.177035    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"
Oct 31 01:45:11.177224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.177052    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld" event={"ID":"89ee38a2-7a31-49ca-b903-af33df249cfa","Type":"ContainerDied","Data":"b1a13934fb0676463765e397422ff8ec57ef71ecca0049b3e06201894810fdbb"}
Oct 31 01:45:11.177224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.177129    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld" event={"ID":"89ee38a2-7a31-49ca-b903-af33df249cfa","Type":"ContainerDied","Data":"8cf7c213ea585af203e04a1837d602383550173a4a788e1719a63aaef722b70f"}
Oct 31 01:45:11.177224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.177151    4476 scope.go:122] "RemoveContainer" containerID="b1a13934fb0676463765e397422ff8ec57ef71ecca0049b3e06201894810fdbb"
Oct 31 01:45:11.187696 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.187681    4476 scope.go:122] "RemoveContainer" containerID="b1a13934fb0676463765e397422ff8ec57ef71ecca0049b3e06201894810fdbb"
Oct 31 01:45:11.187984 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:45:11.187966    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a13934fb0676463765e397422ff8ec57ef71ecca0049b3e06201894810fdbb\": container with ID starting with b1a13934fb0676463765e397422ff8ec57ef71ecca0049b3e06201894810fdbb not found: ID does not exist" containerID="b1a13934fb0676463765e397422ff8ec57ef71ecca0049b3e06201894810fdbb"
Oct 31 01:45:11.188024 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.187994    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a13934fb0676463765e397422ff8ec57ef71ecca0049b3e06201894810fdbb"} err="failed to get container status \"b1a13934fb0676463765e397422ff8ec57ef71ecca0049b3e06201894810fdbb\": rpc error: code = NotFound desc = could not find container \"b1a13934fb0676463765e397422ff8ec57ef71ecca0049b3e06201894810fdbb\": container with ID starting with b1a13934fb0676463765e397422ff8ec57ef71ecca0049b3e06201894810fdbb not found: ID does not exist"
Oct 31 01:45:11.339958 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.339932    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"]
Oct 31 01:45:11.342258 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.342239    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-66100b-67848bd4d5-qr6ld"]
Oct 31 01:45:11.706233 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.706204    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"]
Oct 31 01:45:11.706700 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.706687    4476 cpu_manager.go:414] "containerMap: removing container" podUID="89ee38a2-7a31-49ca-b903-af33df249cfa" containerName="maas-api"
Oct 31 01:45:11.706759 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.706701    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ee38a2-7a31-49ca-b903-af33df249cfa" containerName="maas-api"
Oct 31 01:45:11.706793 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.706768    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ee38a2-7a31-49ca-b903-af33df249cfa" containerName="maas-api"
Oct 31 01:45:11.711208 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.711163    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"
Oct 31 01:45:11.763059 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.763019    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:45:11.766393 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.766368    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"]
Oct 31 01:45:11.812671 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.812644    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mc86\" (UniqueName: \"kubernetes.io/projected/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-kube-api-access-5mc86\") pod \"maas-api-revoke-sub-3bb96c45-fxm6h\" (UID: \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"
Oct 31 01:45:11.812803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.812690    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb96c45-fxm6h\" (UID: \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"
Oct 31 01:45:11.812803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.812751    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb96c45-fxm6h\" (UID: \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"
Oct 31 01:45:11.914315 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.914288    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mc86\" (UniqueName: \"kubernetes.io/projected/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-kube-api-access-5mc86\") pod \"maas-api-revoke-sub-3bb96c45-fxm6h\" (UID: \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"
Oct 31 01:45:11.914507 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.914337    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb96c45-fxm6h\" (UID: \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"
Oct 31 01:45:11.914507 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.914365    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb96c45-fxm6h\" (UID: \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"
Oct 31 01:45:11.914981 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.914959    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb96c45-fxm6h\" (UID: \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"
Oct 31 01:45:11.916782 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:11.916760    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb96c45-fxm6h\" (UID: \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"
Oct 31 01:45:12.050112 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:12.050087    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mc86\" (UniqueName: \"kubernetes.io/projected/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-kube-api-access-5mc86\") pod \"maas-api-revoke-sub-3bb96c45-fxm6h\" (UID: \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"
Oct 31 01:45:12.324846 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:12.324765    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"
Oct 31 01:45:12.453349 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:12.453318    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89ee38a2-7a31-49ca-b903-af33df249cfa" path="/var/lib/kubelet/pods/89ee38a2-7a31-49ca-b903-af33df249cfa/volumes"
Oct 31 01:45:12.472950 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:12.472900    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:12.701508 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:12.701482    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"]
Oct 31 01:45:12.702939 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:45:12.702911    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc1cc49_0f55_4e0e_9a9e_8419dd417dcc.slice/crio-abf22b9cd9163986a5488913459806808898b031c88722f52c5359793dd03692 WatchSource:0}: Error finding container abf22b9cd9163986a5488913459806808898b031c88722f52c5359793dd03692: Status 404 returned error can't find the container with id abf22b9cd9163986a5488913459806808898b031c88722f52c5359793dd03692
Oct 31 01:45:13.192425 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:13.192394    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h" event={"ID":"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc","Type":"ContainerStarted","Data":"571ed9e7c4d2d95bc7dd6e73811fd97f39305635d24e06a3cba4e1bbc2cdb4e2"}
Oct 31 01:45:13.192799 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:13.192446    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h" event={"ID":"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc","Type":"ContainerStarted","Data":"abf22b9cd9163986a5488913459806808898b031c88722f52c5359793dd03692"}
Oct 31 01:45:13.304540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:13.304487    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h" podStartSLOduration=2.304472226 podStartE2EDuration="2.304472226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:13.301930759 +0000 UTC m=+2922.991813886" watchObservedRunningTime="2026-10-31 01:45:13.304472226 +0000 UTC m=+2922.994355387"
Oct 31 01:45:14.199610 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:14.199576    4476 generic.go:356] "Generic (PLEG): container finished" podID="ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc" containerID="571ed9e7c4d2d95bc7dd6e73811fd97f39305635d24e06a3cba4e1bbc2cdb4e2" exitCode=0
Oct 31 01:45:14.200011 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:14.199669    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h" event={"ID":"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc","Type":"ContainerDied","Data":"571ed9e7c4d2d95bc7dd6e73811fd97f39305635d24e06a3cba4e1bbc2cdb4e2"}
Oct 31 01:45:15.335965 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:15.335939    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"
Oct 31 01:45:15.453706 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:15.453673    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-maas-api-xks-ca\") pod \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\" (UID: \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\") "
Oct 31 01:45:15.453926 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:15.453845    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-maas-api-openshift-ca\") pod \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\" (UID: \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\") "
Oct 31 01:45:15.454026 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:15.453941    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-kube-api-access-5mc86\" (UniqueName: \"kubernetes.io/projected/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-kube-api-access-5mc86\") pod \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\" (UID: \"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc\") "
Oct 31 01:45:15.454289 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:15.454263    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-maas-api-openshift-ca" pod "ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc" (UID: "ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:15.454404 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:15.454385    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-maas-api-openshift-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:15.456512 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:15.456445    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-maas-api-xks-ca" pod "ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc" (UID: "ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:45:15.456788 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:15.456755    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-kube-api-access-5mc86" pod "ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc" (UID: "ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc"). InnerVolumeSpecName "kube-api-access-5mc86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:15.555668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:15.555634    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-maas-api-xks-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:15.555668 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:15.555664    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mc86\" (UniqueName: \"kubernetes.io/projected/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc-kube-api-access-5mc86\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:16.212721 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:16.212694    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"
Oct 31 01:45:16.212925 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:16.212690    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h" event={"ID":"ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc","Type":"ContainerDied","Data":"abf22b9cd9163986a5488913459806808898b031c88722f52c5359793dd03692"}
Oct 31 01:45:16.212925 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:16.212818    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abf22b9cd9163986a5488913459806808898b031c88722f52c5359793dd03692"
Oct 31 01:45:16.448199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:16.448131    4476 scope.go:122] "RemoveContainer" containerID="ace15153dc84bc74749e3d2e342481af2e11b5ecbee6053880240949ed017783"
Oct 31 01:45:17.219538 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:17.219502    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz" event={"ID":"28fbc476-4f37-403e-9bd0-dece71e21460","Type":"ContainerStarted","Data":"3048246b8104f759f704f56b48e6363fc30cbfa9cef42ae193f7cab68a6d60eb"}
Oct 31 01:45:17.331231 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:17.331169    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:17.335760 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:17.335715    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz" podStartSLOduration=17.335703371 podStartE2EDuration="17.335703371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:17.333031239 +0000 UTC m=+2927.022914365" watchObservedRunningTime="2026-10-31 01:45:17.335703371 +0000 UTC m=+2927.025586541"
Oct 31 01:45:18.227320 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:18.227283    4476 generic.go:356] "Generic (PLEG): container finished" podID="28fbc476-4f37-403e-9bd0-dece71e21460" containerID="3048246b8104f759f704f56b48e6363fc30cbfa9cef42ae193f7cab68a6d60eb" exitCode=60
Oct 31 01:45:18.227717 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:18.227338    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz" event={"ID":"28fbc476-4f37-403e-9bd0-dece71e21460","Type":"ContainerDied","Data":"3048246b8104f759f704f56b48e6363fc30cbfa9cef42ae193f7cab68a6d60eb"}
Oct 31 01:45:18.227717 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:18.227406    4476 scope.go:122] "RemoveContainer" containerID="ace15153dc84bc74749e3d2e342481af2e11b5ecbee6053880240949ed017783"
Oct 31 01:45:18.227814 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:18.227790    4476 scope.go:122] "RemoveContainer" containerID="3048246b8104f759f704f56b48e6363fc30cbfa9cef42ae193f7cab68a6d60eb"
Oct 31 01:45:18.228092 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:45:18.228067    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz_odh-ai-gateway-infra(28fbc476-4f37-403e-9bd0-dece71e21460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz" podUID="28fbc476-4f37-403e-9bd0-dece71e21460"
Oct 31 01:45:18.347092 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:18.347061    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"]
Oct 31 01:45:19.389318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:19.389295    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"
Oct 31 01:45:19.395501 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:19.395480    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28fbc476-4f37-403e-9bd0-dece71e21460-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fbc476-4f37-403e-9bd0-dece71e21460-maas-api-service-ca\") pod \"28fbc476-4f37-403e-9bd0-dece71e21460\" (UID: \"28fbc476-4f37-403e-9bd0-dece71e21460\") "
Oct 31 01:45:19.395584 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:19.395519    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fbc476-4f37-403e-9bd0-dece71e21460-kube-api-access-pbqkr\" (UniqueName: \"kubernetes.io/projected/28fbc476-4f37-403e-9bd0-dece71e21460-kube-api-access-pbqkr\") pod \"28fbc476-4f37-403e-9bd0-dece71e21460\" (UID: \"28fbc476-4f37-403e-9bd0-dece71e21460\") "
Oct 31 01:45:19.395786 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:19.395766    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fbc476-4f37-403e-9bd0-dece71e21460-maas-api-service-ca" pod "28fbc476-4f37-403e-9bd0-dece71e21460" (UID: "28fbc476-4f37-403e-9bd0-dece71e21460"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:19.397647 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:19.397625    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fbc476-4f37-403e-9bd0-dece71e21460-kube-api-access-pbqkr" pod "28fbc476-4f37-403e-9bd0-dece71e21460" (UID: "28fbc476-4f37-403e-9bd0-dece71e21460"). InnerVolumeSpecName "kube-api-access-pbqkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:19.497160 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:19.496981    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fbc476-4f37-403e-9bd0-dece71e21460-maas-api-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:19.497160 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:19.497010    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbqkr\" (UniqueName: \"kubernetes.io/projected/28fbc476-4f37-403e-9bd0-dece71e21460-kube-api-access-pbqkr\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:20.244691 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:20.244657    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"
Oct 31 01:45:20.244846 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:20.244688    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz" event={"ID":"28fbc476-4f37-403e-9bd0-dece71e21460","Type":"ContainerDied","Data":"8e09aaa19c6586458de6bae3d96c482597992f418a6521c1874f87d382fea9cd"}
Oct 31 01:45:20.244846 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:20.244747    4476 scope.go:122] "RemoveContainer" containerID="3048246b8104f759f704f56b48e6363fc30cbfa9cef42ae193f7cab68a6d60eb"
Oct 31 01:45:20.408909 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:20.408876    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"]
Oct 31 01:45:20.456778 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:20.456748    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-ade881-29890185-q2vkz"]
Oct 31 01:45:21.914311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:21.914271    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:22.085072 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.085040    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:22.453342 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.453310    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28fbc476-4f37-403e-9bd0-dece71e21460" path="/var/lib/kubelet/pods/28fbc476-4f37-403e-9bd0-dece71e21460/volumes"
Oct 31 01:45:22.904872 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.904839    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"]
Oct 31 01:45:22.905355 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.905338    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc" containerName="revoke-keys"
Oct 31 01:45:22.905405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.905356    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc" containerName="revoke-keys"
Oct 31 01:45:22.905405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.905395    4476 cpu_manager.go:414] "containerMap: removing container" podUID="28fbc476-4f37-403e-9bd0-dece71e21460" containerName="cleanup"
Oct 31 01:45:22.905405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.905400    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fbc476-4f37-403e-9bd0-dece71e21460" containerName="cleanup"
Oct 31 01:45:22.905513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.905414    4476 cpu_manager.go:414] "containerMap: removing container" podUID="28fbc476-4f37-403e-9bd0-dece71e21460" containerName="cleanup"
Oct 31 01:45:22.905513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.905423    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fbc476-4f37-403e-9bd0-dece71e21460" containerName="cleanup"
Oct 31 01:45:22.905513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.905438    4476 cpu_manager.go:414] "containerMap: removing container" podUID="28fbc476-4f37-403e-9bd0-dece71e21460" containerName="cleanup"
Oct 31 01:45:22.905513 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.905443    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fbc476-4f37-403e-9bd0-dece71e21460" containerName="cleanup"
Oct 31 01:45:22.905644 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.905526    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fbc476-4f37-403e-9bd0-dece71e21460" containerName="cleanup"
Oct 31 01:45:22.905644 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.905543    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc" containerName="revoke-keys"
Oct 31 01:45:22.905644 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.905553    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fbc476-4f37-403e-9bd0-dece71e21460" containerName="cleanup"
Oct 31 01:45:22.910252 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.910237    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:22.932409 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.932383    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:22.932720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.932421    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:22.932720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.932446    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:22.932720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.932498    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrc9r\" (UniqueName: \"kubernetes.io/projected/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-kube-api-access-mrc9r\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:22.932720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.932568    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:22.932720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.932639    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:22.932720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.932716    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:22.932917 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.932773    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:22.932917 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.932891    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:22.969055 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:22.969030    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"]
Oct 31 01:45:23.014141 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.014115    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-97b317-openshift-default-dockercfg-4lml6\""
Oct 31 01:45:23.033878 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.033853    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.034015 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.033954    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.034015 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.033978    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.034222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.034014    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.034222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.034040    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.034222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.034060    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrc9r\" (UniqueName: \"kubernetes.io/projected/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-kube-api-access-mrc9r\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.034222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.034083    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.034222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.034112    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.034496 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.034223    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.034496 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.034311    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.034496 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.034453    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.034787 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.034766    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.034894 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.034766    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.034951 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.034926    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.036702 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.036675    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.037035 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.037014    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.094289 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.094262    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrc9r\" (UniqueName: \"kubernetes.io/projected/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-kube-api-access-mrc9r\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.094628 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.094601    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.223239 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.223153    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:23.656036 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.656007    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"]
Oct 31 01:45:23.657271 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:45:23.657242    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e9c87c_1fca_4fe9_b2cc_b849f3357ae6.slice/crio-878c15c8030c9c00a74c2f132a8f6cb3af1b75d25c8879fdb2926954f413379b WatchSource:0}: Error finding container 878c15c8030c9c00a74c2f132a8f6cb3af1b75d25c8879fdb2926954f413379b: Status 404 returned error can't find the container with id 878c15c8030c9c00a74c2f132a8f6cb3af1b75d25c8879fdb2926954f413379b
Oct 31 01:45:23.659679 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.659644    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:45:23.659776 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.659712    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:45:23.659776 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:23.659739    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:45:24.270163 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:24.270124    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw" event={"ID":"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6","Type":"ContainerStarted","Data":"a6c6deb09b78dca2b5c2b9b47095551406eccf5b2f1c04742c9fd2193837083f"}
Oct 31 01:45:24.270626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:24.270200    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw" event={"ID":"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6","Type":"ContainerStarted","Data":"878c15c8030c9c00a74c2f132a8f6cb3af1b75d25c8879fdb2926954f413379b"}
Oct 31 01:45:24.391438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:24.391380    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw" podStartSLOduration=2.391365409 podStartE2EDuration="2.391365409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:24.389480707 +0000 UTC m=+2934.079363833" watchObservedRunningTime="2026-10-31 01:45:24.391365409 +0000 UTC m=+2934.081248536"
Oct 31 01:45:25.223980 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:25.223941    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:25.225805 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:25.225782    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:45:25.225881 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:25.225836    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw" podUID="57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 01:45:26.223880 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:26.223847    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:45:26.224321 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:26.223907    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw" podUID="57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 01:45:27.176306 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:27.176215    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:27.224350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:27.224314    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:45:27.224779 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:27.224382    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw" podUID="57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 01:45:28.223735 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:28.223698    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:45:28.223920 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:28.223759    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw" podUID="57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 01:45:29.207383 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:45:29.207329    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:45:29.223840 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:29.223801    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:45:29.223993 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:29.223864    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw" podUID="57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 01:45:30.224605 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:30.224564    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:45:30.225020 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:30.224649    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw" podUID="57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 01:45:31.228790 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:31.228763    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:31.229234 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:31.229200    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:31.229866 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:31.229848    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:45:31.997333 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:31.997301    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:34.901483 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:34.901454    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:35.094483 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:35.094447    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:36.697314 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:36.697280    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:41.512140 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:41.512105    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:45.555296 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:45.555264    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:45.727616 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:45.727583    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:45.940337 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:45.940255    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:46.259510 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:46.259416    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:50.979564 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:50.979526    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:55.665707 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:45:55.665661    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:59.181283 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:45:59.181247    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:46:00.381385 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:00.381349    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:01.392588 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:01.392556    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:01.822058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:01.822028    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:02.001966 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:02.001939    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:05.954423 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:05.954393    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:06.161673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:06.161642    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:06.345444 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:06.345408    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:07.015050 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:07.015018    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:08.219797 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.219764    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"]
Oct 31 01:46:08.220336 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.220322    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fbc476-4f37-403e-9bd0-dece71e21460" containerName="cleanup"
Oct 31 01:46:08.223594 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.223578    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.269312 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.269288    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"]
Oct 31 01:46:08.272857 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.272842    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.352578 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.352545    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.352768 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.352603    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.352768 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.352690    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.352879 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.352772    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.352879 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.352799    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxhvb\" (UniqueName: \"kubernetes.io/projected/69028e29-6686-4cd4-bf5a-94446d2c70d2-kube-api-access-bxhvb\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.352879 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.352869    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk6sv\" (UniqueName: \"kubernetes.io/projected/79b7fba4-9eff-4f65-a9d5-a7625f98253b-kube-api-access-qk6sv\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.353023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.352897    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.353023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.352930    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.353023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.352993    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.353193 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.353049    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.353193 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.353103    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.353193 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.353134    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.353334 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.353242    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.353334 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.353276    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.353334 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.353303    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69028e29-6686-4cd4-bf5a-94446d2c70d2-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.353459 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.353419    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.353518 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.353483    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.353576 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.353519    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.355655 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.355631    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"]
Oct 31 01:46:08.417927 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.417898    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"]
Oct 31 01:46:08.454148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.454116    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.454148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.454147    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69028e29-6686-4cd4-bf5a-94446d2c70d2-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.454397 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.454211    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.454397 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.454257    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.454397 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.454303    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.454397 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.454364    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.454608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.454400    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.454608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.454431    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.454608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.454499    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.454608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.454523    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxhvb\" (UniqueName: \"kubernetes.io/projected/69028e29-6686-4cd4-bf5a-94446d2c70d2-kube-api-access-bxhvb\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.454608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.454560    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk6sv\" (UniqueName: \"kubernetes.io/projected/79b7fba4-9eff-4f65-a9d5-a7625f98253b-kube-api-access-qk6sv\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.454608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.454590    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.454608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.454600    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.454977 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.454958    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.455049 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.455031    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.455399 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.455361    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.455511 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.455410    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.455511 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.455438    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.455511 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.455464    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.455511 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.455489    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.455742 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.455527    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.455742 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.455587    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.455742 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.455599    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.455910 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.455832    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.455910 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.455864    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.455910 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.455879    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69028e29-6686-4cd4-bf5a-94446d2c70d2-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.456086 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.455953    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.456444 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.456424    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.458042 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.457987    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.458193 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.458156    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.458291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.458275    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.458369 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.458329    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.511547 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.511490    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.511719 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.511700    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxhvb\" (UniqueName: \"kubernetes.io/projected/69028e29-6686-4cd4-bf5a-94446d2c70d2-kube-api-access-bxhvb\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.512080 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.512063    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk6sv\" (UniqueName: \"kubernetes.io/projected/79b7fba4-9eff-4f65-a9d5-a7625f98253b-kube-api-access-qk6sv\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.512161 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.512081    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.537116 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.537090    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:08.585264 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.585233    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:08.964644 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.964613    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"]
Oct 31 01:46:08.966159 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:46:08.966136    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b7fba4_9eff_4f65_a9d5_a7625f98253b.slice/crio-6ae581ea13a0ffa93b7480d197bb503a027d26689f70ccb3adce13236bbeb20b WatchSource:0}: Error finding container 6ae581ea13a0ffa93b7480d197bb503a027d26689f70ccb3adce13236bbeb20b: Status 404 returned error can't find the container with id 6ae581ea13a0ffa93b7480d197bb503a027d26689f70ccb3adce13236bbeb20b
Oct 31 01:46:08.968696 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.968661    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:46:08.968786 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.968749    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:46:08.968850 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:08.968784    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:46:09.018778 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:09.018751    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"]
Oct 31 01:46:09.022777 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:46:09.022752    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69028e29_6686_4cd4_bf5a_94446d2c70d2.slice/crio-8c3daed49e430ce658d2ed370f5313429087ee93094a71102a91ef7ca60cbb0f WatchSource:0}: Error finding container 8c3daed49e430ce658d2ed370f5313429087ee93094a71102a91ef7ca60cbb0f: Status 404 returned error can't find the container with id 8c3daed49e430ce658d2ed370f5313429087ee93094a71102a91ef7ca60cbb0f
Oct 31 01:46:09.025241 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:09.025206    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:46:09.025344 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:09.025281    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:46:09.025344 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:09.025325    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:46:09.554608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:09.554566    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc" event={"ID":"69028e29-6686-4cd4-bf5a-94446d2c70d2","Type":"ContainerStarted","Data":"8da72dbc6bd118df10edc74b9e44e2d0f14b3bc02c77627c072c23e0ad6a1329"}
Oct 31 01:46:09.557444 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:09.557413    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc" event={"ID":"69028e29-6686-4cd4-bf5a-94446d2c70d2","Type":"ContainerStarted","Data":"8c3daed49e430ce658d2ed370f5313429087ee93094a71102a91ef7ca60cbb0f"}
Oct 31 01:46:09.558660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:09.558622    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2" event={"ID":"79b7fba4-9eff-4f65-a9d5-a7625f98253b","Type":"ContainerStarted","Data":"ce86eeaa06784847a7fce6b6f34a45e4ec69fc70274fd76aa4ca12ae2313c06b"}
Oct 31 01:46:09.558812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:09.558694    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2" event={"ID":"79b7fba4-9eff-4f65-a9d5-a7625f98253b","Type":"ContainerStarted","Data":"6ae581ea13a0ffa93b7480d197bb503a027d26689f70ccb3adce13236bbeb20b"}
Oct 31 01:46:09.594199 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:09.589310    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:09.699201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:09.699075    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc" podStartSLOduration=1.699054649 podStartE2EDuration="1.699054649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:09.69297769 +0000 UTC m=+2979.382860818" watchObservedRunningTime="2026-10-31 01:46:09.699054649 +0000 UTC m=+2979.388937777"
Oct 31 01:46:09.818263 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:09.817448    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2" podStartSLOduration=1.81742727 podStartE2EDuration="1.81742727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:09.814316186 +0000 UTC m=+2979.504199313" watchObservedRunningTime="2026-10-31 01:46:09.81742727 +0000 UTC m=+2979.507310399"
Oct 31 01:46:10.538347 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:10.538313    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:10.539789 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:10.539770    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:46:10.539871 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:10.539813    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2" podUID="79b7fba4-9eff-4f65-a9d5-a7625f98253b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 01:46:10.610863 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:10.610708    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 01:46:10.610863 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:10.610765    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc" podUID="69028e29-6686-4cd4-bf5a-94446d2c70d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 01:46:11.325095 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:11.323756    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:11.537612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:11.537579    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:46:11.537803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:11.537637    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2" podUID="79b7fba4-9eff-4f65-a9d5-a7625f98253b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 01:46:11.590257 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:11.589343    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 01:46:11.590257 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:11.589400    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc" podUID="69028e29-6686-4cd4-bf5a-94446d2c70d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 01:46:12.537934 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:12.537894    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:46:12.538349 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:12.537951    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2" podUID="79b7fba4-9eff-4f65-a9d5-a7625f98253b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 01:46:12.589604 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:12.589564    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 01:46:12.589896 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:12.589870    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc" podUID="69028e29-6686-4cd4-bf5a-94446d2c70d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 01:46:13.010280 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:13.010190    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:13.190789 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:13.190754    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:13.377752 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:13.377715    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:13.538343 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:13.538307    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:46:13.538705 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:13.538356    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2" podUID="79b7fba4-9eff-4f65-a9d5-a7625f98253b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 01:46:13.586609 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:13.586573    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 01:46:13.586873 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:13.586845    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc" podUID="69028e29-6686-4cd4-bf5a-94446d2c70d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 01:46:14.538154 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:14.538116    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:46:14.538367 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:14.538195    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2" podUID="79b7fba4-9eff-4f65-a9d5-a7625f98253b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 01:46:14.589200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:14.588161    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 01:46:14.589200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:14.588247    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc" podUID="69028e29-6686-4cd4-bf5a-94446d2c70d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 01:46:15.538436 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:15.538402    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:46:15.538839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:15.538453    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2" podUID="79b7fba4-9eff-4f65-a9d5-a7625f98253b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 01:46:15.586471 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:15.586437    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 01:46:15.586573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:15.586499    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc" podUID="69028e29-6686-4cd4-bf5a-94446d2c70d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 01:46:16.242889 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:16.242850    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:16.542156 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:16.542128    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:16.542555 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:16.542499    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:16.543259 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:16.543240    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:46:16.589459 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:16.589439    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:16.589719 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:16.589703    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:16.590408 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:16.590393    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 01:46:20.609464 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:20.609433    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:20.797639 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:20.797605    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:22.336135 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:22.336107    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:22.519332 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:22.519303    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:23.005589 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:23.005557    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:26.042899 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:26.042870    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:26.213525 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:26.213485    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:26.766998 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:26.766942    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:28.459032 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:28.458998    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:29.182613 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:46:29.182558    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:46:29.189679 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:29.189646    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:29.386329 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:29.386293    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:29.574411 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:29.574375    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:30.470983 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:30.470949    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:30.760142 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:30.760058    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:30.815229 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:30.815201    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:46:30.846589 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:30.846564    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:46:31.400381 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:31.400349    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:33.484812 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:33.484776    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:37.219648 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:37.219614    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:37.403898 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:37.403859    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:37.525701 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:37.525620    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"]
Oct 31 01:46:37.525868 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:37.525814    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr" podUID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" containerName="istio-proxy" containerID="cri-o://b3ff90d7279d38c25ce3100dde332cb15d213fa786bda0d162a5fbc86eb4d7ee" gracePeriod=30
Oct 31 01:46:37.577039 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:37.577011    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:38.489112 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:38.489077    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:42.796616 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.796538    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:46:42.813311 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.813282    4476 generic.go:356] "Generic (PLEG): container finished" podID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" containerID="b3ff90d7279d38c25ce3100dde332cb15d213fa786bda0d162a5fbc86eb4d7ee" exitCode=0
Oct 31 01:46:42.813441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.813329    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"
Oct 31 01:46:42.813441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.813369    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr" event={"ID":"209964ea-0fcd-4a91-9f58-ad393cbc0b8a","Type":"ContainerDied","Data":"b3ff90d7279d38c25ce3100dde332cb15d213fa786bda0d162a5fbc86eb4d7ee"}
Oct 31 01:46:42.813441 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.813421    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr" event={"ID":"209964ea-0fcd-4a91-9f58-ad393cbc0b8a","Type":"ContainerDied","Data":"c49628903482bd04bde644363fea36301344f5783ca1093e831d61302a4f623e"}
Oct 31 01:46:42.813559 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.813447    4476 scope.go:122] "RemoveContainer" containerID="b3ff90d7279d38c25ce3100dde332cb15d213fa786bda0d162a5fbc86eb4d7ee"
Oct 31 01:46:42.822583 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.822564    4476 scope.go:122] "RemoveContainer" containerID="71dd802f7093920c7be9c2374d4842ec1d51274ad0157918be83b59e4ce5c173"
Oct 31 01:46:42.833652 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.833631    4476 scope.go:122] "RemoveContainer" containerID="b3ff90d7279d38c25ce3100dde332cb15d213fa786bda0d162a5fbc86eb4d7ee"
Oct 31 01:46:42.833949 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:46:42.833920    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ff90d7279d38c25ce3100dde332cb15d213fa786bda0d162a5fbc86eb4d7ee\": container with ID starting with b3ff90d7279d38c25ce3100dde332cb15d213fa786bda0d162a5fbc86eb4d7ee not found: ID does not exist" containerID="b3ff90d7279d38c25ce3100dde332cb15d213fa786bda0d162a5fbc86eb4d7ee"
Oct 31 01:46:42.834032 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.833950    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ff90d7279d38c25ce3100dde332cb15d213fa786bda0d162a5fbc86eb4d7ee"} err="failed to get container status \"b3ff90d7279d38c25ce3100dde332cb15d213fa786bda0d162a5fbc86eb4d7ee\": rpc error: code = NotFound desc = could not find container \"b3ff90d7279d38c25ce3100dde332cb15d213fa786bda0d162a5fbc86eb4d7ee\": container with ID starting with b3ff90d7279d38c25ce3100dde332cb15d213fa786bda0d162a5fbc86eb4d7ee not found: ID does not exist"
Oct 31 01:46:42.834032 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.833971    4476 scope.go:122] "RemoveContainer" containerID="71dd802f7093920c7be9c2374d4842ec1d51274ad0157918be83b59e4ce5c173"
Oct 31 01:46:42.834313 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:46:42.834291    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71dd802f7093920c7be9c2374d4842ec1d51274ad0157918be83b59e4ce5c173\": container with ID starting with 71dd802f7093920c7be9c2374d4842ec1d51274ad0157918be83b59e4ce5c173 not found: ID does not exist" containerID="71dd802f7093920c7be9c2374d4842ec1d51274ad0157918be83b59e4ce5c173"
Oct 31 01:46:42.834366 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.834323    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dd802f7093920c7be9c2374d4842ec1d51274ad0157918be83b59e4ce5c173"} err="failed to get container status \"71dd802f7093920c7be9c2374d4842ec1d51274ad0157918be83b59e4ce5c173\": rpc error: code = NotFound desc = could not find container \"71dd802f7093920c7be9c2374d4842ec1d51274ad0157918be83b59e4ce5c173\": container with ID starting with 71dd802f7093920c7be9c2374d4842ec1d51274ad0157918be83b59e4ce5c173 not found: ID does not exist"
Oct 31 01:46:42.886025 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.885992    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-credential-socket\") pod \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") "
Oct 31 01:46:42.886868 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.886840    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-workload-socket\") pod \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") "
Oct 31 01:46:42.887016 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.886883    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-podinfo\") pod \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") "
Oct 31 01:46:42.887016 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.886943    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-envoy\") pod \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") "
Oct 31 01:46:42.887138 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.887019    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-workload-certs\") pod \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") "
Oct 31 01:46:42.887138 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.887110    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-token\" (UniqueName: \"kubernetes.io/projected/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-token\") pod \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") "
Oct 31 01:46:42.887305 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.887169    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istiod-ca-cert\") pod \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") "
Oct 31 01:46:42.889636 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.889592    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-kube-api-access-4t9zb\" (UniqueName: \"kubernetes.io/projected/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-kube-api-access-4t9zb\") pod \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") "
Oct 31 01:46:42.889792 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.889708    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-data\") pod \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\" (UID: \"209964ea-0fcd-4a91-9f58-ad393cbc0b8a\") "
Oct 31 01:46:42.891321 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.886487    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-credential-socket" pod "209964ea-0fcd-4a91-9f58-ad393cbc0b8a" (UID: "209964ea-0fcd-4a91-9f58-ad393cbc0b8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:46:42.891321 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.890810    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-data" pod "209964ea-0fcd-4a91-9f58-ad393cbc0b8a" (UID: "209964ea-0fcd-4a91-9f58-ad393cbc0b8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:46:42.891321 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.891130    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-workload-socket" pod "209964ea-0fcd-4a91-9f58-ad393cbc0b8a" (UID: "209964ea-0fcd-4a91-9f58-ad393cbc0b8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:46:42.907210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.894199    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-workload-certs" pod "209964ea-0fcd-4a91-9f58-ad393cbc0b8a" (UID: "209964ea-0fcd-4a91-9f58-ad393cbc0b8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:46:42.907210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.894444    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istiod-ca-cert" pod "209964ea-0fcd-4a91-9f58-ad393cbc0b8a" (UID: "209964ea-0fcd-4a91-9f58-ad393cbc0b8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:46:42.911286 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.910446    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-envoy" pod "209964ea-0fcd-4a91-9f58-ad393cbc0b8a" (UID: "209964ea-0fcd-4a91-9f58-ad393cbc0b8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:46:42.911418 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.911385    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-kube-api-access-4t9zb" pod "209964ea-0fcd-4a91-9f58-ad393cbc0b8a" (UID: "209964ea-0fcd-4a91-9f58-ad393cbc0b8a"). InnerVolumeSpecName "kube-api-access-4t9zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:46:42.911741 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.911622    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-token" pod "209964ea-0fcd-4a91-9f58-ad393cbc0b8a" (UID: "209964ea-0fcd-4a91-9f58-ad393cbc0b8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:46:42.911741 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.911705    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-podinfo" pod "209964ea-0fcd-4a91-9f58-ad393cbc0b8a" (UID: "209964ea-0fcd-4a91-9f58-ad393cbc0b8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:46:42.992494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.992297    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:46:42.992494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.992338    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:46:42.992494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.992356    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:46:42.992494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.992370    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:46:42.992494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.992385    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:46:42.992494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.992399    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:46:42.992494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.992413    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9zb\" (UniqueName: \"kubernetes.io/projected/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-kube-api-access-4t9zb\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:46:42.992494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.992430    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:46:42.992494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:42.992444    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209964ea-0fcd-4a91-9f58-ad393cbc0b8a-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:46:43.398163 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:43.398074    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"]
Oct 31 01:46:43.403720 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:43.403679    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-001a26-openshift-default-79d9c75c5c-5r7pr"]
Oct 31 01:46:43.407216 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:43.406249    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:44.457726 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:44.457694    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" path="/var/lib/kubelet/pods/209964ea-0fcd-4a91-9f58-ad393cbc0b8a/volumes"
Oct 31 01:46:46.285014 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:46.284973    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:47.081522 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:47.081459    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:47.382350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:47.382238    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:48.173085 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:48.173051    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:52.807073 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:52.807036    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:57.588576 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:57.588531    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:57.770220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:57.770151    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:58.343159 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:58.343121    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:58.572328 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:46:58.572290    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:59.181728 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:46:59.181695    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:47:00.983308 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:00.983278    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:01.627960 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:01.627925    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:01.820097 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:01.820060    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:02.006769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:02.006680    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:02.419395 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:02.419360    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:02.617320 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:02.617291    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:02.818519 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:02.818483    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:03.133784 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:03.133699    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:05.829934 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:05.829899    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:06.004320 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:06.004282    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:06.438131 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:06.438065    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:06.642917 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:06.642882    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:07.531594 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:07.531562    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:10.672803 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:10.672771    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:10.844982 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:10.844948    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:10.992786 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:10.992716    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"]
Oct 31 01:47:10.993290 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:10.993276    4476 cpu_manager.go:414] "containerMap: removing container" podUID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" containerName="istio-proxy"
Oct 31 01:47:10.993339 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:10.993293    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" containerName="istio-proxy"
Oct 31 01:47:10.993450 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:10.993441    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" containerName="istio-proxy"
Oct 31 01:47:10.996638 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:10.996625    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"
Oct 31 01:47:11.051986 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:11.051955    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"]
Oct 31 01:47:11.118793 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:11.118761    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-b992fe-dockercfg-fmzzl\""
Oct 31 01:47:11.124194 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:11.119288    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-b992fe\""
Oct 31 01:47:11.128044 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:11.128009    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f52bd\" (UniqueName: \"kubernetes.io/projected/ae3e94c6-c7e3-4e24-a832-4e632806f191-kube-api-access-f52bd\") pod \"payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t\" (UID: \"ae3e94c6-c7e3-4e24-a832-4e632806f191\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"
Oct 31 01:47:11.128343 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:11.128318    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3e94c6-c7e3-4e24-a832-4e632806f191-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t\" (UID: \"ae3e94c6-c7e3-4e24-a832-4e632806f191\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"
Oct 31 01:47:11.230042 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:11.230010    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3e94c6-c7e3-4e24-a832-4e632806f191-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t\" (UID: \"ae3e94c6-c7e3-4e24-a832-4e632806f191\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"
Oct 31 01:47:11.230234 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:11.230067    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f52bd\" (UniqueName: \"kubernetes.io/projected/ae3e94c6-c7e3-4e24-a832-4e632806f191-kube-api-access-f52bd\") pod \"payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t\" (UID: \"ae3e94c6-c7e3-4e24-a832-4e632806f191\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"
Oct 31 01:47:11.230694 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:11.230672    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3e94c6-c7e3-4e24-a832-4e632806f191-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t\" (UID: \"ae3e94c6-c7e3-4e24-a832-4e632806f191\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"
Oct 31 01:47:11.287340 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:11.287309    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f52bd\" (UniqueName: \"kubernetes.io/projected/ae3e94c6-c7e3-4e24-a832-4e632806f191-kube-api-access-f52bd\") pod \"payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t\" (UID: \"ae3e94c6-c7e3-4e24-a832-4e632806f191\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"
Oct 31 01:47:11.309999 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:11.309974    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"
Oct 31 01:47:11.684900 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:11.684870    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"]
Oct 31 01:47:11.686406 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:47:11.686382    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3e94c6_c7e3_4e24_a832_4e632806f191.slice/crio-aefc44844575e98516061b262d449056b4e18af81242b0a1357f487be891ac3e WatchSource:0}: Error finding container aefc44844575e98516061b262d449056b4e18af81242b0a1357f487be891ac3e: Status 404 returned error can't find the container with id aefc44844575e98516061b262d449056b4e18af81242b0a1357f487be891ac3e
Oct 31 01:47:12.064158 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:12.064125    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t" event={"ID":"ae3e94c6-c7e3-4e24-a832-4e632806f191","Type":"ContainerStarted","Data":"5be17606205e439b003c9ad57e27e9fa13724ae73f3fa75f94cea6a476385c42"}
Oct 31 01:47:12.064373 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:12.064196    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t" event={"ID":"ae3e94c6-c7e3-4e24-a832-4e632806f191","Type":"ContainerStarted","Data":"aefc44844575e98516061b262d449056b4e18af81242b0a1357f487be891ac3e"}
Oct 31 01:47:12.064452 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:12.064393    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"
Oct 31 01:47:12.178410 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:12.178349    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t" podStartSLOduration=2.17832811 podStartE2EDuration="2.17832811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:12.176967858 +0000 UTC m=+3041.866850988" watchObservedRunningTime="2026-10-31 01:47:12.17832811 +0000 UTC m=+3041.868211237"
Oct 31 01:47:12.773864 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:12.773810    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:17.885018 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:17.884959    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:22.525108 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:22.525078    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:22.715318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:22.715286    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:23.077965 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:23.077928    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"
Oct 31 01:47:23.969569 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:23.969531    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"]
Oct 31 01:47:23.970063 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:23.970048    4476 cpu_manager.go:414] "containerMap: removing container" podUID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" containerName="istio-proxy"
Oct 31 01:47:23.970108 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:23.970065    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" containerName="istio-proxy"
Oct 31 01:47:23.970198 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:23.970167    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="209964ea-0fcd-4a91-9f58-ad393cbc0b8a" containerName="istio-proxy"
Oct 31 01:47:23.973533 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:23.973518    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.060453 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.060423    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.060594 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.060461    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.060594 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.060494    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.060594 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.060549    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.060594 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.060582    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.060738 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.060649    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.060738 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.060702    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmbgr\" (UniqueName: \"kubernetes.io/projected/c76cc5c8-d8e3-44d4-b419-9a38439489e2-kube-api-access-zmbgr\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.060820 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.060802    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.060859 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.060837    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.078573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.078535    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"]
Oct 31 01:47:24.161854 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.161823    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.162036 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.161868    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.162036 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.161895    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.162036 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.161940    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmbgr\" (UniqueName: \"kubernetes.io/projected/c76cc5c8-d8e3-44d4-b419-9a38439489e2-kube-api-access-zmbgr\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.162036 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.162029    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.162291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.162069    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.162291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.162211    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.162291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.162249    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.162291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.162254    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.162506 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.162213    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.162506 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.162301    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.162506 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.162465    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.162677 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.162520    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.162923 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.162902    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.164607 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.164581    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.164702 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.164645    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.218429 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.218404    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.218523 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.218497    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmbgr\" (UniqueName: \"kubernetes.io/projected/c76cc5c8-d8e3-44d4-b419-9a38439489e2-kube-api-access-zmbgr\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.287047 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.287024    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:24.714291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.714266    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"]
Oct 31 01:47:24.715250 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:47:24.715224    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76cc5c8_d8e3_44d4_b419_9a38439489e2.slice/crio-0c15bc6627c98c5c3188b10468f403c36a61e1b2d93ad7106eb9122391a910d9 WatchSource:0}: Error finding container 0c15bc6627c98c5c3188b10468f403c36a61e1b2d93ad7106eb9122391a910d9: Status 404 returned error can't find the container with id 0c15bc6627c98c5c3188b10468f403c36a61e1b2d93ad7106eb9122391a910d9
Oct 31 01:47:24.717522 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.717489    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:47:24.717605 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.717554    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:47:24.717605 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:24.717591    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:47:25.152830 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:25.152794    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5" event={"ID":"c76cc5c8-d8e3-44d4-b419-9a38439489e2","Type":"ContainerStarted","Data":"dc856bc84de5955739c7a4380ead588c2a668b05e3de384469724bd7e5954dea"}
Oct 31 01:47:25.153291 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:25.152854    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5" event={"ID":"c76cc5c8-d8e3-44d4-b419-9a38439489e2","Type":"ContainerStarted","Data":"0c15bc6627c98c5c3188b10468f403c36a61e1b2d93ad7106eb9122391a910d9"}
Oct 31 01:47:25.284224 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:25.281030    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5" podStartSLOduration=2.281010299 podStartE2EDuration="2.281010299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:25.278734798 +0000 UTC m=+3054.968617926" watchObservedRunningTime="2026-10-31 01:47:25.281010299 +0000 UTC m=+3054.970893427"
Oct 31 01:47:25.287231 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:25.287198    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:25.288927 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:25.288902    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 01:47:25.289039 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:25.288950    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5" podUID="c76cc5c8-d8e3-44d4-b419-9a38439489e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 01:47:26.287913 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:26.287877    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 01:47:26.288374 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:26.287935    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5" podUID="c76cc5c8-d8e3-44d4-b419-9a38439489e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 01:47:26.390066 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:26.389999    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:26.841251 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:26.841199    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:27.184817 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:27.184722    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:27.287707 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:27.287660    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 01:47:27.287922 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:27.287730    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5" podUID="c76cc5c8-d8e3-44d4-b419-9a38439489e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 01:47:28.288045 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:28.288006    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 01:47:28.288477 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:28.288058    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5" podUID="c76cc5c8-d8e3-44d4-b419-9a38439489e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 01:47:28.307465 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:28.307436    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:28.520551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:28.520492    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:28.701681 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:28.701596    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:29.207423 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:47:29.207392    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:47:29.287840 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:29.287804    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 01:47:29.288010 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:29.287874    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5" podUID="c76cc5c8-d8e3-44d4-b419-9a38439489e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 01:47:29.292843 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:29.292815    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:30.156114 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:30.156084    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:30.287717 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:30.287671    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 01:47:30.287899 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:30.287743    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5" podUID="c76cc5c8-d8e3-44d4-b419-9a38439489e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 01:47:30.769261 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:30.769224    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:31.045775 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:31.045709    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:31.287593 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:31.287561    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 01:47:31.287797 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:31.287613    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5" podUID="c76cc5c8-d8e3-44d4-b419-9a38439489e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 01:47:32.300160 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:32.300132    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:32.300631 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:32.300608    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:32.301944 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:32.301924    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 01:47:33.514005 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:33.513945    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:35.026405 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:35.026371    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:35.221082 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:35.221048    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:35.408737 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:35.408689    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:37.871350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:37.871298    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:38.642129 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:38.642099    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:38.818932 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:38.818899    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:39.537696 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:39.537663    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:39.752075 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:39.752039    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:43.834938 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:43.834895    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:44.012741 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:44.012708    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:44.186042 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:44.185958    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:46.614661 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:46.614628    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:49.245118 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:49.245086    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:54.251226 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:54.251153    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:54.433795 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:54.433760    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:54.613961 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:54.613929    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:58.237470 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:58.237437    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:58.416488 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:58.416459    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:59.180502 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:47:59.180466    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:47:59.686117 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:47:59.686089    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:02.088495 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:02.088463    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:04.484002 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:04.483963    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:09.288383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:09.288346    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:14.086354 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:14.086257    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:18.476971 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.476941    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"]
Oct 31 01:48:18.481264 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.481244    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:18.533320 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.533298    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-dc9e63-serving-cert\""
Oct 31 01:48:18.537073 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.537052    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"]
Oct 31 01:48:18.624928 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.624885    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/810203c9-65fd-4958-a46e-7ecf24a22f41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:18.625106 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.624947    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqs5m\" (UniqueName: \"kubernetes.io/projected/810203c9-65fd-4958-a46e-7ecf24a22f41-kube-api-access-qqs5m\") pod \"maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:18.625460 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.625427    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:18.625606 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.625508    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:18.713645 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.713616    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:18.731042 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.730927    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:18.731042 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.731029    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:18.731322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.731218    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/810203c9-65fd-4958-a46e-7ecf24a22f41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:18.731322 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.731257    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqs5m\" (UniqueName: \"kubernetes.io/projected/810203c9-65fd-4958-a46e-7ecf24a22f41-kube-api-access-qqs5m\") pod \"maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:18.732985 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:48:18.732235    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-serving-cert: secret "maas-api-e2e-shared-a-w2-dc9e63-serving-cert" not found
Oct 31 01:48:18.732985 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:48:18.732320    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-maas-api-tls podName:810203c9-65fd-4958-a46e-7ecf24a22f41 nodeName:}" failed. No retries permitted until 2026-10-31 01:48:19.232295123 +0000 UTC m=+3108.922178274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-maas-api-tls") pod "maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc" (UID: "810203c9-65fd-4958-a46e-7ecf24a22f41") : secret "maas-api-e2e-shared-a-w2-dc9e63-serving-cert" not found
Oct 31 01:48:18.733403 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.733367    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/810203c9-65fd-4958-a46e-7ecf24a22f41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:18.743370 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.738844    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:18.788282 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.788232    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqs5m\" (UniqueName: \"kubernetes.io/projected/810203c9-65fd-4958-a46e-7ecf24a22f41-kube-api-access-qqs5m\") pod \"maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:18.888827 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:18.888756    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:19.082285 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:19.082259    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:19.237874 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:19.237848    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:19.240436 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:19.240412    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:19.394920 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:19.394849    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:19.768749 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:19.768724    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"]
Oct 31 01:48:19.770476 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:48:19.770423    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810203c9_65fd_4958_a46e_7ecf24a22f41.slice/crio-f3b064611f4ee174a4b533c1e02827fac7e1badbd73d102ef326cf9b778b51ca WatchSource:0}: Error finding container f3b064611f4ee174a4b533c1e02827fac7e1badbd73d102ef326cf9b778b51ca: Status 404 returned error can't find the container with id f3b064611f4ee174a4b533c1e02827fac7e1badbd73d102ef326cf9b778b51ca
Oct 31 01:48:19.772120 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:19.772099    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:48:20.482657 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:20.482622    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc" event={"ID":"810203c9-65fd-4958-a46e-7ecf24a22f41","Type":"ContainerStarted","Data":"f3b064611f4ee174a4b533c1e02827fac7e1badbd73d102ef326cf9b778b51ca"}
Oct 31 01:48:21.489434 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:21.489394    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc" event={"ID":"810203c9-65fd-4958-a46e-7ecf24a22f41","Type":"ContainerStarted","Data":"8ccd014fade3ed542bff2d40e43fc915a8c252377232c75d1d9c7dd11d3d309c"}
Oct 31 01:48:21.489858 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:21.489470    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:21.604014 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:21.603966    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc" podStartSLOduration=2.754110817 podStartE2EDuration="3.603953534s" totalImagesPullingTime="849.842717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:21.601914635 +0000 UTC m=+3111.291797764" watchObservedRunningTime="2026-10-31 01:48:21.603953534 +0000 UTC m=+3111.293836725"
Oct 31 01:48:22.987060 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:22.987022    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:23.605301 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:23.605245    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:27.500709 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:27.500682    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 01:48:28.375463 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:28.375430    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:29.181533 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:48:29.181495    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:48:32.727518 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:32.727482    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:33.096330 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:33.096279    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:33.673430 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:33.673397    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:34.017457 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:34.017380    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:37.969164 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:37.969121    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:38.309989 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:38.309950    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:38.512625 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:38.512596    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:39.739532 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:39.739494    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:40.168506 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:40.168477    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:41.022058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:41.022026    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:41.373117 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:41.373090    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:43.155346 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.155310    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:43.348624 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.348571    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:43.531092 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.531047    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:43.667162 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.667131    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"]
Oct 31 01:48:43.671049 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.671034    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"
Oct 31 01:48:43.725037 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.725008    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c382414a-6f3f-41fc-b773-b315da186244-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz\" (UID: \"c382414a-6f3f-41fc-b773-b315da186244\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"
Oct 31 01:48:43.725037 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.725038    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rbsf\" (UniqueName: \"kubernetes.io/projected/c382414a-6f3f-41fc-b773-b315da186244-kube-api-access-5rbsf\") pod \"payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz\" (UID: \"c382414a-6f3f-41fc-b773-b315da186244\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"
Oct 31 01:48:43.773384 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.773352    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-b6f24b\""
Oct 31 01:48:43.773626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.773382    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-b6f24b-dockercfg-nfw7h\""
Oct 31 01:48:43.777889 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.777858    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"]
Oct 31 01:48:43.826482 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.826400    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c382414a-6f3f-41fc-b773-b315da186244-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz\" (UID: \"c382414a-6f3f-41fc-b773-b315da186244\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"
Oct 31 01:48:43.826482 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.826429    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rbsf\" (UniqueName: \"kubernetes.io/projected/c382414a-6f3f-41fc-b773-b315da186244-kube-api-access-5rbsf\") pod \"payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz\" (UID: \"c382414a-6f3f-41fc-b773-b315da186244\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"
Oct 31 01:48:43.827166 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.827140    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c382414a-6f3f-41fc-b773-b315da186244-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz\" (UID: \"c382414a-6f3f-41fc-b773-b315da186244\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"
Oct 31 01:48:43.883976 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.883944    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rbsf\" (UniqueName: \"kubernetes.io/projected/c382414a-6f3f-41fc-b773-b315da186244-kube-api-access-5rbsf\") pod \"payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz\" (UID: \"c382414a-6f3f-41fc-b773-b315da186244\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"
Oct 31 01:48:43.984732 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:43.984701    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"
Oct 31 01:48:44.360734 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:44.360702    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"]
Oct 31 01:48:44.363369 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:48:44.363343    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc382414a_6f3f_41fc_b773_b315da186244.slice/crio-2a7442e900e2f9a7cc9a8b1194e1e24391b517457eda7db7d47103854a7f914d WatchSource:0}: Error finding container 2a7442e900e2f9a7cc9a8b1194e1e24391b517457eda7db7d47103854a7f914d: Status 404 returned error can't find the container with id 2a7442e900e2f9a7cc9a8b1194e1e24391b517457eda7db7d47103854a7f914d
Oct 31 01:48:44.618433 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:44.618355    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz" event={"ID":"c382414a-6f3f-41fc-b773-b315da186244","Type":"ContainerStarted","Data":"04df0c872c39388d4d7df2ae9d83a592f483a50319eb9ba6910b27fea4143a42"}
Oct 31 01:48:44.618433 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:44.618409    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz" event={"ID":"c382414a-6f3f-41fc-b773-b315da186244","Type":"ContainerStarted","Data":"2a7442e900e2f9a7cc9a8b1194e1e24391b517457eda7db7d47103854a7f914d"}
Oct 31 01:48:44.618611 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:44.618445    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"
Oct 31 01:48:44.731626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:44.731572    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz" podStartSLOduration=1.731554687 podStartE2EDuration="1.731554687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:44.730416232 +0000 UTC m=+3134.420299361" watchObservedRunningTime="2026-10-31 01:48:44.731554687 +0000 UTC m=+3134.421437816"
Oct 31 01:48:44.958528 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:44.958449    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:45.149836 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:45.149796    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:45.342711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:45.342681    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:46.970084 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:46.970050    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:48.153047 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.153018    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:48.346129 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.346097    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:48.528084 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.528063    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"]
Oct 31 01:48:48.529806 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.529778    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:48.532396 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.532381    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"
Oct 31 01:48:48.587864 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.587838    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"]
Oct 31 01:48:48.588315 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.588287    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94n9j\" (UniqueName: \"kubernetes.io/projected/0beaabcf-d87c-4769-9a19-84e9124568a3-kube-api-access-94n9j\") pod \"payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5\" (UID: \"0beaabcf-d87c-4769-9a19-84e9124568a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"
Oct 31 01:48:48.588479 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.588460    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0beaabcf-d87c-4769-9a19-84e9124568a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5\" (UID: \"0beaabcf-d87c-4769-9a19-84e9124568a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"
Oct 31 01:48:48.633328 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.633296    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-083e17-dockercfg-6ncj5\""
Oct 31 01:48:48.633328 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.633313    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-083e17\""
Oct 31 01:48:48.686011 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.685983    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"]
Oct 31 01:48:48.689640 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.689594    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0beaabcf-d87c-4769-9a19-84e9124568a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5\" (UID: \"0beaabcf-d87c-4769-9a19-84e9124568a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"
Oct 31 01:48:48.689829 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.689810    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94n9j\" (UniqueName: \"kubernetes.io/projected/0beaabcf-d87c-4769-9a19-84e9124568a3-kube-api-access-94n9j\") pod \"payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5\" (UID: \"0beaabcf-d87c-4769-9a19-84e9124568a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"
Oct 31 01:48:48.690424 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.690399    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0beaabcf-d87c-4769-9a19-84e9124568a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5\" (UID: \"0beaabcf-d87c-4769-9a19-84e9124568a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"
Oct 31 01:48:48.690694 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.690680    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"
Oct 31 01:48:48.739210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.739163    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"]
Oct 31 01:48:48.753629 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.753601    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94n9j\" (UniqueName: \"kubernetes.io/projected/0beaabcf-d87c-4769-9a19-84e9124568a3-kube-api-access-94n9j\") pod \"payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5\" (UID: \"0beaabcf-d87c-4769-9a19-84e9124568a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"
Oct 31 01:48:48.791464 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.791388    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll4jz\" (UniqueName: \"kubernetes.io/projected/b14aee9c-3186-4e2a-9c84-9b50cb844194-kube-api-access-ll4jz\") pod \"payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2\" (UID: \"b14aee9c-3186-4e2a-9c84-9b50cb844194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"
Oct 31 01:48:48.791606 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.791579    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b14aee9c-3186-4e2a-9c84-9b50cb844194-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2\" (UID: \"b14aee9c-3186-4e2a-9c84-9b50cb844194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"
Oct 31 01:48:48.845333 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.845311    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"
Oct 31 01:48:48.892376 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.892341    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b14aee9c-3186-4e2a-9c84-9b50cb844194-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2\" (UID: \"b14aee9c-3186-4e2a-9c84-9b50cb844194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"
Oct 31 01:48:48.892550 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.892477    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll4jz\" (UniqueName: \"kubernetes.io/projected/b14aee9c-3186-4e2a-9c84-9b50cb844194-kube-api-access-ll4jz\") pod \"payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2\" (UID: \"b14aee9c-3186-4e2a-9c84-9b50cb844194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"
Oct 31 01:48:48.893148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.893122    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b14aee9c-3186-4e2a-9c84-9b50cb844194-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2\" (UID: \"b14aee9c-3186-4e2a-9c84-9b50cb844194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"
Oct 31 01:48:48.949633 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:48.949602    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll4jz\" (UniqueName: \"kubernetes.io/projected/b14aee9c-3186-4e2a-9c84-9b50cb844194-kube-api-access-ll4jz\") pod \"payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2\" (UID: \"b14aee9c-3186-4e2a-9c84-9b50cb844194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"
Oct 31 01:48:49.004863 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:49.004832    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"
Oct 31 01:48:49.221515 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:49.221489    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"]
Oct 31 01:48:49.222278 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:48:49.222247    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0beaabcf_d87c_4769_9a19_84e9124568a3.slice/crio-93bb376fccc4224b9a3458b14381562b09fdb8d12d3c3a524fa5c3435e231dc3 WatchSource:0}: Error finding container 93bb376fccc4224b9a3458b14381562b09fdb8d12d3c3a524fa5c3435e231dc3: Status 404 returned error can't find the container with id 93bb376fccc4224b9a3458b14381562b09fdb8d12d3c3a524fa5c3435e231dc3
Oct 31 01:48:49.381882 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:49.381858    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"]
Oct 31 01:48:49.382960 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:48:49.382933    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14aee9c_3186_4e2a_9c84_9b50cb844194.slice/crio-24f33506d4f1ed9449900f3fe72d4091d5e2479dcebd50ad2d34254ede343e4d WatchSource:0}: Error finding container 24f33506d4f1ed9449900f3fe72d4091d5e2479dcebd50ad2d34254ede343e4d: Status 404 returned error can't find the container with id 24f33506d4f1ed9449900f3fe72d4091d5e2479dcebd50ad2d34254ede343e4d
Oct 31 01:48:49.650770 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:49.650681    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2" event={"ID":"b14aee9c-3186-4e2a-9c84-9b50cb844194","Type":"ContainerStarted","Data":"1be306d582c8a005b4d5b7b5b35bea6d795a4252fe9e349058bf69bd182c3931"}
Oct 31 01:48:49.650770 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:49.650747    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"
Oct 31 01:48:49.650972 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:49.650772    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2" event={"ID":"b14aee9c-3186-4e2a-9c84-9b50cb844194","Type":"ContainerStarted","Data":"24f33506d4f1ed9449900f3fe72d4091d5e2479dcebd50ad2d34254ede343e4d"}
Oct 31 01:48:49.652377 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:49.652346    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5" event={"ID":"0beaabcf-d87c-4769-9a19-84e9124568a3","Type":"ContainerStarted","Data":"a12a183319309ba46cbff46237de987a256480ec08cb386f9c3c25dad1b2e884"}
Oct 31 01:48:49.652510 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:49.652403    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"
Oct 31 01:48:49.652510 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:49.652427    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5" event={"ID":"0beaabcf-d87c-4769-9a19-84e9124568a3","Type":"ContainerStarted","Data":"93bb376fccc4224b9a3458b14381562b09fdb8d12d3c3a524fa5c3435e231dc3"}
Oct 31 01:48:49.766141 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:49.766093    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2" podStartSLOduration=1.766080908 podStartE2EDuration="1.766080908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:49.764377489 +0000 UTC m=+3139.454260616" watchObservedRunningTime="2026-10-31 01:48:49.766080908 +0000 UTC m=+3139.455964034"
Oct 31 01:48:49.884271 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:49.884216    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5" podStartSLOduration=1.884200534 podStartE2EDuration="1.884200534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:49.883881075 +0000 UTC m=+3139.573764203" watchObservedRunningTime="2026-10-31 01:48:49.884200534 +0000 UTC m=+3139.574083700"
Oct 31 01:48:53.053838 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:53.053798    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:53.416916 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:53.416832    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:53.587769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:53.587736    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:54.138113 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:54.138082    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:54.492783 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:54.492702    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:55.627325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:55.627291    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"
Oct 31 01:48:58.159091 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:58.159062    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:58.358287 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:58.358253    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:58.546523 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:48:58.546493    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:59.181487 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:48:59.181452    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:49:00.661225 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:00.661168    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"
Oct 31 01:49:00.661744 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:00.661393    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"
Oct 31 01:49:01.426705 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:01.426672    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:01.623249 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:01.616586    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:01.958520 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:01.958488    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"]
Oct 31 01:49:01.962383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:01.962368    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"
Oct 31 01:49:02.018565 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.018543    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7fc058\""
Oct 31 01:49:02.051010 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.050975    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efb0ead-026d-472a-b36d-8ad0cccd3487-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l\" (UID: \"5efb0ead-026d-472a-b36d-8ad0cccd3487\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"
Oct 31 01:49:02.051193 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.051026    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2wpf\" (UniqueName: \"kubernetes.io/projected/5efb0ead-026d-472a-b36d-8ad0cccd3487-kube-api-access-h2wpf\") pod \"payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l\" (UID: \"5efb0ead-026d-472a-b36d-8ad0cccd3487\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"
Oct 31 01:49:02.067938 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.067913    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7fc058-dockercfg-jvzm7\""
Oct 31 01:49:02.070869 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.070847    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"]
Oct 31 01:49:02.152669 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.152640    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2wpf\" (UniqueName: \"kubernetes.io/projected/5efb0ead-026d-472a-b36d-8ad0cccd3487-kube-api-access-h2wpf\") pod \"payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l\" (UID: \"5efb0ead-026d-472a-b36d-8ad0cccd3487\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"
Oct 31 01:49:02.152828 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.152813    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efb0ead-026d-472a-b36d-8ad0cccd3487-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l\" (UID: \"5efb0ead-026d-472a-b36d-8ad0cccd3487\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"
Oct 31 01:49:02.153501 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.153484    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efb0ead-026d-472a-b36d-8ad0cccd3487-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l\" (UID: \"5efb0ead-026d-472a-b36d-8ad0cccd3487\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"
Oct 31 01:49:02.209100 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.209043    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2wpf\" (UniqueName: \"kubernetes.io/projected/5efb0ead-026d-472a-b36d-8ad0cccd3487-kube-api-access-h2wpf\") pod \"payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l\" (UID: \"5efb0ead-026d-472a-b36d-8ad0cccd3487\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"
Oct 31 01:49:02.275785 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.275761    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"
Oct 31 01:49:02.499273 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.499141    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:02.648037 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.648010    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"]
Oct 31 01:49:02.649613 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:49:02.649583    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efb0ead_026d_472a_b36d_8ad0cccd3487.slice/crio-aca7e8f4ed873e2296b404a58f237eed13bd6b07a47401ac82b07cc0c02281ea WatchSource:0}: Error finding container aca7e8f4ed873e2296b404a58f237eed13bd6b07a47401ac82b07cc0c02281ea: Status 404 returned error can't find the container with id aca7e8f4ed873e2296b404a58f237eed13bd6b07a47401ac82b07cc0c02281ea
Oct 31 01:49:02.739367 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.739331    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l" event={"ID":"5efb0ead-026d-472a-b36d-8ad0cccd3487","Type":"ContainerStarted","Data":"705593f2bb7d51b9ea6b91231dc51de412afd5fac74f7d82b86c82f74dbc8217"}
Oct 31 01:49:02.739508 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.739399    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l" event={"ID":"5efb0ead-026d-472a-b36d-8ad0cccd3487","Type":"ContainerStarted","Data":"aca7e8f4ed873e2296b404a58f237eed13bd6b07a47401ac82b07cc0c02281ea"}
Oct 31 01:49:02.739573 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.739552    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"
Oct 31 01:49:02.811845 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.811822    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"]
Oct 31 01:49:02.812030 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.812011    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr" podUID="50cd6adf-c72d-497e-bed0-c64642bb8dea" containerName="maas-api" containerID="cri-o://ff833358e023aeeef54c4ba4e559cd5886ede3dbdc3c08965b46faa33466437c" gracePeriod=30
Oct 31 01:49:02.852533 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.852488    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l" podStartSLOduration=1.852474445 podStartE2EDuration="1.852474445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:02.85137291 +0000 UTC m=+3152.541256042" watchObservedRunningTime="2026-10-31 01:49:02.852474445 +0000 UTC m=+3152.542357590"
Oct 31 01:49:02.944712 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:02.944667    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:03.058910 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.058887    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:49:03.163187 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.163098    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50cd6adf-c72d-497e-bed0-c64642bb8dea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cd6adf-c72d-497e-bed0-c64642bb8dea-openshift-service-ca\") pod \"50cd6adf-c72d-497e-bed0-c64642bb8dea\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") "
Oct 31 01:49:03.163187 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.163137    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-metrics-certs\") pod \"50cd6adf-c72d-497e-bed0-c64642bb8dea\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") "
Oct 31 01:49:03.163388 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.163196    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-maas-api-tls\") pod \"50cd6adf-c72d-497e-bed0-c64642bb8dea\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") "
Oct 31 01:49:03.163388 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.163244    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cd6adf-c72d-497e-bed0-c64642bb8dea-kube-api-access-74xtz\" (UniqueName: \"kubernetes.io/projected/50cd6adf-c72d-497e-bed0-c64642bb8dea-kube-api-access-74xtz\") pod \"50cd6adf-c72d-497e-bed0-c64642bb8dea\" (UID: \"50cd6adf-c72d-497e-bed0-c64642bb8dea\") "
Oct 31 01:49:03.163519 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.163499    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cd6adf-c72d-497e-bed0-c64642bb8dea-openshift-service-ca" pod "50cd6adf-c72d-497e-bed0-c64642bb8dea" (UID: "50cd6adf-c72d-497e-bed0-c64642bb8dea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:03.163782 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.163762    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50cd6adf-c72d-497e-bed0-c64642bb8dea-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:03.165511 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.165486    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cd6adf-c72d-497e-bed0-c64642bb8dea-kube-api-access-74xtz" pod "50cd6adf-c72d-497e-bed0-c64642bb8dea" (UID: "50cd6adf-c72d-497e-bed0-c64642bb8dea"). InnerVolumeSpecName "kube-api-access-74xtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:03.165612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.165522    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-metrics-certs" pod "50cd6adf-c72d-497e-bed0-c64642bb8dea" (UID: "50cd6adf-c72d-497e-bed0-c64642bb8dea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:49:03.165612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.165533    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-maas-api-tls" pod "50cd6adf-c72d-497e-bed0-c64642bb8dea" (UID: "50cd6adf-c72d-497e-bed0-c64642bb8dea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:49:03.265171 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.265141    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:03.265171 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.265165    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50cd6adf-c72d-497e-bed0-c64642bb8dea-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:03.265362 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.265200    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74xtz\" (UniqueName: \"kubernetes.io/projected/50cd6adf-c72d-497e-bed0-c64642bb8dea-kube-api-access-74xtz\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:03.747663 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.747624    4476 generic.go:356] "Generic (PLEG): container finished" podID="50cd6adf-c72d-497e-bed0-c64642bb8dea" containerID="ff833358e023aeeef54c4ba4e559cd5886ede3dbdc3c08965b46faa33466437c" exitCode=0
Oct 31 01:49:03.747841 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.747661    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr" event={"ID":"50cd6adf-c72d-497e-bed0-c64642bb8dea","Type":"ContainerDied","Data":"ff833358e023aeeef54c4ba4e559cd5886ede3dbdc3c08965b46faa33466437c"}
Oct 31 01:49:03.747841 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.747723    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr" event={"ID":"50cd6adf-c72d-497e-bed0-c64642bb8dea","Type":"ContainerDied","Data":"af263559aaeb3667adec4c9a67317086286d133c8f916a30be868d6f100ad716"}
Oct 31 01:49:03.747841 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.747752    4476 scope.go:122] "RemoveContainer" containerID="ff833358e023aeeef54c4ba4e559cd5886ede3dbdc3c08965b46faa33466437c"
Oct 31 01:49:03.747841 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.747675    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"
Oct 31 01:49:03.758743 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.758728    4476 scope.go:122] "RemoveContainer" containerID="ff833358e023aeeef54c4ba4e559cd5886ede3dbdc3c08965b46faa33466437c"
Oct 31 01:49:03.759041 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:49:03.759023    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff833358e023aeeef54c4ba4e559cd5886ede3dbdc3c08965b46faa33466437c\": container with ID starting with ff833358e023aeeef54c4ba4e559cd5886ede3dbdc3c08965b46faa33466437c not found: ID does not exist" containerID="ff833358e023aeeef54c4ba4e559cd5886ede3dbdc3c08965b46faa33466437c"
Oct 31 01:49:03.759085 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.759049    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff833358e023aeeef54c4ba4e559cd5886ede3dbdc3c08965b46faa33466437c"} err="failed to get container status \"ff833358e023aeeef54c4ba4e559cd5886ede3dbdc3c08965b46faa33466437c\": rpc error: code = NotFound desc = could not find container \"ff833358e023aeeef54c4ba4e559cd5886ede3dbdc3c08965b46faa33466437c\": container with ID starting with ff833358e023aeeef54c4ba4e559cd5886ede3dbdc3c08965b46faa33466437c not found: ID does not exist"
Oct 31 01:49:03.915517 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.915488    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"]
Oct 31 01:49:03.964779 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:03.964748    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-25b9ff-68669479c8-9dcbr"]
Oct 31 01:49:04.454425 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:04.454385    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50cd6adf-c72d-497e-bed0-c64642bb8dea" path="/var/lib/kubelet/pods/50cd6adf-c72d-497e-bed0-c64642bb8dea/volumes"
Oct 31 01:49:04.612876 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:04.612846    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:04.814937 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:04.814904    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:04.997612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:04.997577    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:05.558541 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:05.558511    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:09.915633 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:09.915595    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:10.472922 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:10.472888    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"]
Oct 31 01:49:10.580456 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:10.580402    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:13.751540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:13.751512    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"
Oct 31 01:49:15.075697 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:15.075664    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:16.877189 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:16.877142    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:18.233629 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.233596    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"]
Oct 31 01:49:18.234276 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.234258    4476 cpu_manager.go:414] "containerMap: removing container" podUID="50cd6adf-c72d-497e-bed0-c64642bb8dea" containerName="maas-api"
Oct 31 01:49:18.234276 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.234277    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cd6adf-c72d-497e-bed0-c64642bb8dea" containerName="maas-api"
Oct 31 01:49:18.234404 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.234394    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cd6adf-c72d-497e-bed0-c64642bb8dea" containerName="maas-api"
Oct 31 01:49:18.240145 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.240121    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.300915 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.300871    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"]
Oct 31 01:49:18.332850 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.332817    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbb5\" (UniqueName: \"kubernetes.io/projected/069276a8-4b8e-4db0-9dd1-86c7b402309d-kube-api-access-kxbb5\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.333034 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.332992    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069276a8-4b8e-4db0-9dd1-86c7b402309d-istiod-ca-cert\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.333101 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.333085    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-credential-socket\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.333146 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.333121    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-token\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.333260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.333246    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-workload-certs\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.333301 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.333286    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-podinfo\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.333341 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.333319    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-workload-socket\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.333403 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.333377    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-data\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.333459 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.333444    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-envoy\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.346419 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.346394    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-9c9c3c-openshift-default-dockercfg-b48gt\""
Oct 31 01:49:18.434763 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.434722    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069276a8-4b8e-4db0-9dd1-86c7b402309d-istiod-ca-cert\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.434939 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.434799    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-credential-socket\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.434939 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.434834    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-token\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.434939 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.434897    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-workload-certs\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.434939 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.434930    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-podinfo\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.435220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.434961    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-workload-socket\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.435220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.434983    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-data\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.435220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.435029    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-envoy\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.435220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.435102    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbb5\" (UniqueName: \"kubernetes.io/projected/069276a8-4b8e-4db0-9dd1-86c7b402309d-kube-api-access-kxbb5\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.435456 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.435268    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-credential-socket\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.435456 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.435415    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-workload-certs\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.435537 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.435460    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-workload-socket\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.435578 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.435539    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-data\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.435616 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.435607    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069276a8-4b8e-4db0-9dd1-86c7b402309d-istiod-ca-cert\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.437579 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.437562    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-podinfo\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.437630 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.437560    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-envoy\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.492098 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.492031    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-token\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.492572 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.492104    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbb5\" (UniqueName: \"kubernetes.io/projected/069276a8-4b8e-4db0-9dd1-86c7b402309d-kube-api-access-kxbb5\") pod \"e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") " pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.556197 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.556141    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:18.989200 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.989155    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"]
Oct 31 01:49:18.990077 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:49:18.990042    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069276a8_4b8e_4db0_9dd1_86c7b402309d.slice/crio-f16eb21e971a32c55f916639c1cd9869a3ff707229b0dcc9c3a55e3c95a34f81 WatchSource:0}: Error finding container f16eb21e971a32c55f916639c1cd9869a3ff707229b0dcc9c3a55e3c95a34f81: Status 404 returned error can't find the container with id f16eb21e971a32c55f916639c1cd9869a3ff707229b0dcc9c3a55e3c95a34f81
Oct 31 01:49:18.992581 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.992547    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:49:18.992654 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.992613    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:49:18.992654 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:18.992641    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:49:19.473791 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:19.473766    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:19.666377 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:19.666347    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:19.852799 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:19.852766    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt" event={"ID":"069276a8-4b8e-4db0-9dd1-86c7b402309d","Type":"ContainerStarted","Data":"fb6ebf7571a9c70935bbe989502cc41023bd102e899e9361385a4fe02dae4de3"}
Oct 31 01:49:19.852974 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:19.852821    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt" event={"ID":"069276a8-4b8e-4db0-9dd1-86c7b402309d","Type":"ContainerStarted","Data":"f16eb21e971a32c55f916639c1cd9869a3ff707229b0dcc9c3a55e3c95a34f81"}
Oct 31 01:49:20.063581 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:20.063523    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt" podStartSLOduration=2.063510443 podStartE2EDuration="2.063510443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:20.062907621 +0000 UTC m=+3169.752790751" watchObservedRunningTime="2026-10-31 01:49:20.063510443 +0000 UTC m=+3169.753393570"
Oct 31 01:49:20.163715 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:20.163627    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:20.556735 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:20.556707    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:20.562612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:20.562590    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:20.859103 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:20.859029    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:20.860074 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:20.860055    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:49:22.955722 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:22.955691    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:23.145491 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:23.145461    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:23.347499 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:23.347472    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:25.412223 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.412193    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:25.493958 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.493912    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p" podUID="da4721b0-ca4f-4538-829d-0590b02ccfdf" containerName="main" containerID="cri-o://4d25fe49d772658d7ac221c8cae5c0db430f6137192fdb4930784a0220dfdc60" gracePeriod=45
Oct 31 01:49:25.755142 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.755120    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:49:25.893938 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.893903    4476 generic.go:356] "Generic (PLEG): container finished" podID="da4721b0-ca4f-4538-829d-0590b02ccfdf" containerID="4d25fe49d772658d7ac221c8cae5c0db430f6137192fdb4930784a0220dfdc60" exitCode=0
Oct 31 01:49:25.894124 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.893968    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"
Oct 31 01:49:25.894124 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.893969    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p" event={"ID":"da4721b0-ca4f-4538-829d-0590b02ccfdf","Type":"ContainerDied","Data":"4d25fe49d772658d7ac221c8cae5c0db430f6137192fdb4930784a0220dfdc60"}
Oct 31 01:49:25.894124 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.894059    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p" event={"ID":"da4721b0-ca4f-4538-829d-0590b02ccfdf","Type":"ContainerDied","Data":"6dad1f9098652450d93fa8a0b20b47110e35a6cd980a1aea78df7203027b908b"}
Oct 31 01:49:25.894124 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.894082    4476 scope.go:122] "RemoveContainer" containerID="4d25fe49d772658d7ac221c8cae5c0db430f6137192fdb4930784a0220dfdc60"
Oct 31 01:49:25.904434 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.904412    4476 scope.go:122] "RemoveContainer" containerID="4d25fe49d772658d7ac221c8cae5c0db430f6137192fdb4930784a0220dfdc60"
Oct 31 01:49:25.904761 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:49:25.904740    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d25fe49d772658d7ac221c8cae5c0db430f6137192fdb4930784a0220dfdc60\": container with ID starting with 4d25fe49d772658d7ac221c8cae5c0db430f6137192fdb4930784a0220dfdc60 not found: ID does not exist" containerID="4d25fe49d772658d7ac221c8cae5c0db430f6137192fdb4930784a0220dfdc60"
Oct 31 01:49:25.904873 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.904768    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d25fe49d772658d7ac221c8cae5c0db430f6137192fdb4930784a0220dfdc60"} err="failed to get container status \"4d25fe49d772658d7ac221c8cae5c0db430f6137192fdb4930784a0220dfdc60\": rpc error: code = NotFound desc = could not find container \"4d25fe49d772658d7ac221c8cae5c0db430f6137192fdb4930784a0220dfdc60\": container with ID starting with 4d25fe49d772658d7ac221c8cae5c0db430f6137192fdb4930784a0220dfdc60 not found: ID does not exist"
Oct 31 01:49:25.916227 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.916157    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-dshm\") pod \"da4721b0-ca4f-4538-829d-0590b02ccfdf\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") "
Oct 31 01:49:25.916346 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.916240    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4721b0-ca4f-4538-829d-0590b02ccfdf-kube-api-access-4rvjm\" (UniqueName: \"kubernetes.io/projected/da4721b0-ca4f-4538-829d-0590b02ccfdf-kube-api-access-4rvjm\") pod \"da4721b0-ca4f-4538-829d-0590b02ccfdf\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") "
Oct 31 01:49:25.916346 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.916306    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-model-cache\") pod \"da4721b0-ca4f-4538-829d-0590b02ccfdf\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") "
Oct 31 01:49:25.916482 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.916448    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-tmp-dir\") pod \"da4721b0-ca4f-4538-829d-0590b02ccfdf\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") "
Oct 31 01:49:25.916482 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.916477    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4721b0-ca4f-4538-829d-0590b02ccfdf-tls-certs\" (UniqueName: \"kubernetes.io/secret/da4721b0-ca4f-4538-829d-0590b02ccfdf-tls-certs\") pod \"da4721b0-ca4f-4538-829d-0590b02ccfdf\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") "
Oct 31 01:49:25.916596 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.916504    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-home\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-home\") pod \"da4721b0-ca4f-4538-829d-0590b02ccfdf\" (UID: \"da4721b0-ca4f-4538-829d-0590b02ccfdf\") "
Oct 31 01:49:25.916596 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.916537    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-model-cache" pod "da4721b0-ca4f-4538-829d-0590b02ccfdf" (UID: "da4721b0-ca4f-4538-829d-0590b02ccfdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:25.916756 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.916737    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-tmp-dir" pod "da4721b0-ca4f-4538-829d-0590b02ccfdf" (UID: "da4721b0-ca4f-4538-829d-0590b02ccfdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:25.917090 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.917065    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:25.917090 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.917087    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:25.917267 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.917086    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-home" pod "da4721b0-ca4f-4538-829d-0590b02ccfdf" (UID: "da4721b0-ca4f-4538-829d-0590b02ccfdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:25.918760 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.918734    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4721b0-ca4f-4538-829d-0590b02ccfdf-kube-api-access-4rvjm" pod "da4721b0-ca4f-4538-829d-0590b02ccfdf" (UID: "da4721b0-ca4f-4538-829d-0590b02ccfdf"). InnerVolumeSpecName "kube-api-access-4rvjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:25.919150 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.919127    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4721b0-ca4f-4538-829d-0590b02ccfdf-tls-certs" pod "da4721b0-ca4f-4538-829d-0590b02ccfdf" (UID: "da4721b0-ca4f-4538-829d-0590b02ccfdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:49:25.919607 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:25.919589    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-dshm" pod "da4721b0-ca4f-4538-829d-0590b02ccfdf" (UID: "da4721b0-ca4f-4538-829d-0590b02ccfdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:26.017795 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.017767    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:26.017795 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.017793    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rvjm\" (UniqueName: \"kubernetes.io/projected/da4721b0-ca4f-4538-829d-0590b02ccfdf-kube-api-access-4rvjm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:26.017997 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.017803    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4721b0-ca4f-4538-829d-0590b02ccfdf-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:26.017997 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.017813    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4721b0-ca4f-4538-829d-0590b02ccfdf-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:26.169520 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.169434    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:26.314860 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.314824    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"]
Oct 31 01:49:26.318871 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.318846    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-25b9ff/praxis-route-25b9ff-kserve-5fc694cd8f-tcs5p"]
Oct 31 01:49:26.350126 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.350099    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:26.455209 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.455125    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4721b0-ca4f-4538-829d-0590b02ccfdf" path="/var/lib/kubelet/pods/da4721b0-ca4f-4538-829d-0590b02ccfdf/volumes"
Oct 31 01:49:26.558853 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.558824    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"]
Oct 31 01:49:26.559342 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.559329    4476 cpu_manager.go:414] "containerMap: removing container" podUID="da4721b0-ca4f-4538-829d-0590b02ccfdf" containerName="main"
Oct 31 01:49:26.559394 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.559343    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4721b0-ca4f-4538-829d-0590b02ccfdf" containerName="main"
Oct 31 01:49:26.559479 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.559470    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4721b0-ca4f-4538-829d-0590b02ccfdf" containerName="main"
Oct 31 01:49:26.564000 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.563985    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"
Oct 31 01:49:26.616482 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.616439    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-9c9c3c\""
Oct 31 01:49:26.616664 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.616622    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-9c9c3c-dockercfg-h65mk\""
Oct 31 01:49:26.624900 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.624568    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"]
Oct 31 01:49:26.726201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.726087    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmcx\" (UniqueName: \"kubernetes.io/projected/047a012c-e995-469b-a04a-ce12cdd56f94-kube-api-access-vvmcx\") pod \"payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh\" (UID: \"047a012c-e995-469b-a04a-ce12cdd56f94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"
Oct 31 01:49:26.726358 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.726336    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047a012c-e995-469b-a04a-ce12cdd56f94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh\" (UID: \"047a012c-e995-469b-a04a-ce12cdd56f94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"
Oct 31 01:49:26.827380 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.827352    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmcx\" (UniqueName: \"kubernetes.io/projected/047a012c-e995-469b-a04a-ce12cdd56f94-kube-api-access-vvmcx\") pod \"payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh\" (UID: \"047a012c-e995-469b-a04a-ce12cdd56f94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"
Oct 31 01:49:26.827566 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.827505    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047a012c-e995-469b-a04a-ce12cdd56f94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh\" (UID: \"047a012c-e995-469b-a04a-ce12cdd56f94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"
Oct 31 01:49:26.828288 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.828262    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047a012c-e995-469b-a04a-ce12cdd56f94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh\" (UID: \"047a012c-e995-469b-a04a-ce12cdd56f94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"
Oct 31 01:49:26.888831 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:26.888806    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmcx\" (UniqueName: \"kubernetes.io/projected/047a012c-e995-469b-a04a-ce12cdd56f94-kube-api-access-vvmcx\") pod \"payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh\" (UID: \"047a012c-e995-469b-a04a-ce12cdd56f94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"
Oct 31 01:49:27.176993 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:27.176958    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"
Oct 31 01:49:27.549830 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:27.549798    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"]
Oct 31 01:49:27.550798 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:49:27.550761    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047a012c_e995_469b_a04a_ce12cdd56f94.slice/crio-8c2babfca644daa7d7b0f1817a178166050d6fe6cd8eb55b28dd53ff3eb45af4 WatchSource:0}: Error finding container 8c2babfca644daa7d7b0f1817a178166050d6fe6cd8eb55b28dd53ff3eb45af4: Status 404 returned error can't find the container with id 8c2babfca644daa7d7b0f1817a178166050d6fe6cd8eb55b28dd53ff3eb45af4
Oct 31 01:49:27.908860 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:27.908772    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh" event={"ID":"047a012c-e995-469b-a04a-ce12cdd56f94","Type":"ContainerStarted","Data":"8962fcdaac8c054844c17945fa927b24920237fae23e67a8644d2afdf19936f6"}
Oct 31 01:49:27.908860 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:27.908831    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh" event={"ID":"047a012c-e995-469b-a04a-ce12cdd56f94","Type":"ContainerStarted","Data":"8c2babfca644daa7d7b0f1817a178166050d6fe6cd8eb55b28dd53ff3eb45af4"}
Oct 31 01:49:27.908860 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:27.908860    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"
Oct 31 01:49:28.022253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:28.022194    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh" podStartSLOduration=2.022164521 podStartE2EDuration="2.022164521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:28.0203578 +0000 UTC m=+3177.710240928" watchObservedRunningTime="2026-10-31 01:49:28.022164521 +0000 UTC m=+3177.712047648"
Oct 31 01:49:29.182569 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:49:29.182532    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:49:30.095351 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.095318    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"]
Oct 31 01:49:30.099341 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.099320    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:30.211366 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.211330    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-b6f24b-serving-cert\""
Oct 31 01:49:30.214426 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.214407    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"]
Oct 31 01:49:30.268403 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.268376    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:30.268555 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.268532    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccd4s\" (UniqueName: \"kubernetes.io/projected/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-kube-api-access-ccd4s\") pod \"maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:30.268603 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.268580    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:30.268762 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.268736    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:30.333300 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.333265    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:30.370090 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.370016    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccd4s\" (UniqueName: \"kubernetes.io/projected/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-kube-api-access-ccd4s\") pod \"maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:30.370090 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.370051    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:30.370358 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.370123    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:30.370358 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.370197    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:30.370904 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.370876    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:30.373047 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.373023    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:30.373208 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.373050    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:30.428030 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.427993    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccd4s\" (UniqueName: \"kubernetes.io/projected/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-kube-api-access-ccd4s\") pod \"maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:30.517295 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.517263    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:30.706947 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.706866    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:30.712830 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:30.712811    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:31.095389 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:31.095361    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"]
Oct 31 01:49:31.096280 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:49:31.096257    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb736102_7fdb_47d2_93f9_fa3fa5d67ed5.slice/crio-fa02d18df262a9d6269835de07a9c54db1792a9617e62367a868c117d8c465f1 WatchSource:0}: Error finding container fa02d18df262a9d6269835de07a9c54db1792a9617e62367a868c117d8c465f1: Status 404 returned error can't find the container with id fa02d18df262a9d6269835de07a9c54db1792a9617e62367a868c117d8c465f1
Oct 31 01:49:31.933310 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:31.933285    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl" event={"ID":"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5","Type":"ContainerStarted","Data":"fa02d18df262a9d6269835de07a9c54db1792a9617e62367a868c117d8c465f1"}
Oct 31 01:49:32.948798 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:32.948757    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl" event={"ID":"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5","Type":"ContainerStarted","Data":"237ef368ef249fa06430b23f6dcad3c042405ce07bb7f3d99056d22a74b54ac0"}
Oct 31 01:49:32.949228 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:32.948850    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:33.066080 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:33.066025    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl" podStartSLOduration=2.282002914 podStartE2EDuration="3.066010533s" totalImagesPullingTime="784.007619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:33.064786603 +0000 UTC m=+3182.754669730" watchObservedRunningTime="2026-10-31 01:49:33.066010533 +0000 UTC m=+3182.755893677"
Oct 31 01:49:35.299678 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:35.299633    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:38.917874 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:38.917842    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"
Oct 31 01:49:38.959947 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:38.959922    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 01:49:39.915395 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:39.915364    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:40.117470 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:40.117431    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:44.795887 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:44.795804    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:46.972088 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:46.972054    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:47.400196 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:47.400137    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:48.278458 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:48.278427    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:49.523649 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:49.523619    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:53.923797 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:53.923767    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:54.110009 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:54.109977    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:54.301633 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:54.301608    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:58.925503 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:58.925469    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"]
Oct 31 01:49:58.930717 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:58.930698    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:49:58.984880 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:58.984853    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"]
Oct 31 01:49:59.009562 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.009523    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:49:59.009710 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.009667    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfffc20-e86c-42da-b9d2-58d24d9152aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:49:59.009814 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.009795    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwgkv\" (UniqueName: \"kubernetes.io/projected/cbfffc20-e86c-42da-b9d2-58d24d9152aa-kube-api-access-qwgkv\") pod \"maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:49:59.009880 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.009838    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:49:59.030525 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.030500    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-7fc058-serving-cert\""
Oct 31 01:49:59.110800 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.110764    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:49:59.110979 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.110844    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfffc20-e86c-42da-b9d2-58d24d9152aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:49:59.110979 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.110921    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwgkv\" (UniqueName: \"kubernetes.io/projected/cbfffc20-e86c-42da-b9d2-58d24d9152aa-kube-api-access-qwgkv\") pod \"maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:49:59.110979 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.110957    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:49:59.111116 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:49:59.111069    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-serving-cert: secret "maas-api-e2e-shared-b-w3-7fc058-serving-cert" not found
Oct 31 01:49:59.111159 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:49:59.111135    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-maas-api-tls podName:cbfffc20-e86c-42da-b9d2-58d24d9152aa nodeName:}" failed. No retries permitted until 2026-10-31 01:49:59.611116105 +0000 UTC m=+3209.300999210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-maas-api-tls") pod "maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52" (UID: "cbfffc20-e86c-42da-b9d2-58d24d9152aa") : secret "maas-api-e2e-shared-b-w3-7fc058-serving-cert" not found
Oct 31 01:49:59.111660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.111607    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfffc20-e86c-42da-b9d2-58d24d9152aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:49:59.113438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.113413    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:49:59.166007 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.165975    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:59.167605 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.167572    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwgkv\" (UniqueName: \"kubernetes.io/projected/cbfffc20-e86c-42da-b9d2-58d24d9152aa-kube-api-access-qwgkv\") pod \"maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:49:59.197362 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:49:59.197275    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:49:59.346880 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.346839    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:59.538075 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.538052    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:59.616436 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.616405    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:49:59.619078 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.619051    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:49:59.844463 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:49:59.844387    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:50:00.220516 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:00.220485    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"]
Oct 31 01:50:00.221412 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:50:00.221386    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfffc20_e86c_42da_b9d2_58d24d9152aa.slice/crio-87f83e7867796a037702332739aa34bd460fc1ec44ce4009e8cccdada6803855 WatchSource:0}: Error finding container 87f83e7867796a037702332739aa34bd460fc1ec44ce4009e8cccdada6803855: Status 404 returned error can't find the container with id 87f83e7867796a037702332739aa34bd460fc1ec44ce4009e8cccdada6803855
Oct 31 01:50:00.688746 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:00.688710    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:01.113497 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:01.113464    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52" event={"ID":"cbfffc20-e86c-42da-b9d2-58d24d9152aa","Type":"ContainerStarted","Data":"87f83e7867796a037702332739aa34bd460fc1ec44ce4009e8cccdada6803855"}
Oct 31 01:50:02.121138 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:02.121102    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52" event={"ID":"cbfffc20-e86c-42da-b9d2-58d24d9152aa","Type":"ContainerStarted","Data":"0ff5a134efcda51384adeaaf97c041a8e4c1eeedf4758586ca77354947dbb863"}
Oct 31 01:50:02.121677 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:02.121212    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:50:02.244302 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:02.244245    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52" podStartSLOduration=3.413345947 podStartE2EDuration="4.244227223s" totalImagesPullingTime="830.881276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:02.241349749 +0000 UTC m=+3211.931232877" watchObservedRunningTime="2026-10-31 01:50:02.244227223 +0000 UTC m=+3211.934110347"
Oct 31 01:50:04.398260 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:04.398229    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:05.982749 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:05.981672    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"]
Oct 31 01:50:06.278964 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:06.278904    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:06.472706 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:06.472652    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:08.132113 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:08.132082    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 01:50:09.313492 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:09.313459    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:10.982593 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:10.982551    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb" podUID="a3defef0-a205-49c8-9bb1-ed3b194b329b" containerName="payload-pre-processing" containerID="cri-o://1f89f0cf0c160d5ad444c1ca8d0f6e0aaef735cdb8de636fef9bd17ed141ce03" gracePeriod=25
Oct 31 01:50:11.179376 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:11.179344    4476 generic.go:356] "Generic (PLEG): container finished" podID="a3defef0-a205-49c8-9bb1-ed3b194b329b" containerID="1f89f0cf0c160d5ad444c1ca8d0f6e0aaef735cdb8de636fef9bd17ed141ce03" exitCode=0
Oct 31 01:50:11.179589 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:11.179485    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb" event={"ID":"a3defef0-a205-49c8-9bb1-ed3b194b329b","Type":"ContainerDied","Data":"1f89f0cf0c160d5ad444c1ca8d0f6e0aaef735cdb8de636fef9bd17ed141ce03"}
Oct 31 01:50:11.255494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:11.255471    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"
Oct 31 01:50:11.351146 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:11.351113    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3defef0-a205-49c8-9bb1-ed3b194b329b-kube-api-access-hpsbj\" (UniqueName: \"kubernetes.io/projected/a3defef0-a205-49c8-9bb1-ed3b194b329b-kube-api-access-hpsbj\") pod \"a3defef0-a205-49c8-9bb1-ed3b194b329b\" (UID: \"a3defef0-a205-49c8-9bb1-ed3b194b329b\") "
Oct 31 01:50:11.351327 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:11.351247    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3defef0-a205-49c8-9bb1-ed3b194b329b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3defef0-a205-49c8-9bb1-ed3b194b329b-plugins-config-volume\") pod \"a3defef0-a205-49c8-9bb1-ed3b194b329b\" (UID: \"a3defef0-a205-49c8-9bb1-ed3b194b329b\") "
Oct 31 01:50:11.351734 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:11.351708    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3defef0-a205-49c8-9bb1-ed3b194b329b-plugins-config-volume" pod "a3defef0-a205-49c8-9bb1-ed3b194b329b" (UID: "a3defef0-a205-49c8-9bb1-ed3b194b329b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:11.351987 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:11.351938    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3defef0-a205-49c8-9bb1-ed3b194b329b-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:11.353406 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:11.353382    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3defef0-a205-49c8-9bb1-ed3b194b329b-kube-api-access-hpsbj" pod "a3defef0-a205-49c8-9bb1-ed3b194b329b" (UID: "a3defef0-a205-49c8-9bb1-ed3b194b329b"). InnerVolumeSpecName "kube-api-access-hpsbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:11.453023 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:11.452991    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpsbj\" (UniqueName: \"kubernetes.io/projected/a3defef0-a205-49c8-9bb1-ed3b194b329b-kube-api-access-hpsbj\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:12.186522 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:12.186492    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"
Oct 31 01:50:12.186522 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:12.186516    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb" event={"ID":"a3defef0-a205-49c8-9bb1-ed3b194b329b","Type":"ContainerDied","Data":"6780788d327e9a94f0631d81d6a8c7b822375d622885c432ffbb341a2eabb114"}
Oct 31 01:50:12.187062 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:12.186575    4476 scope.go:122] "RemoveContainer" containerID="1f89f0cf0c160d5ad444c1ca8d0f6e0aaef735cdb8de636fef9bd17ed141ce03"
Oct 31 01:50:12.354058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:12.354026    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"]
Oct 31 01:50:12.362215 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:12.362193    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ade881-54f84d99c9-xlmdb"]
Oct 31 01:50:12.454841 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:12.454769    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3defef0-a205-49c8-9bb1-ed3b194b329b" path="/var/lib/kubelet/pods/a3defef0-a205-49c8-9bb1-ed3b194b329b/volumes"
Oct 31 01:50:12.885054 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:12.885022    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:13.473115 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:13.473081    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:14.139993 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:14.139957    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:15.047700 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:15.047671    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"]
Oct 31 01:50:15.097021 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:15.096996    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb96c45-fxm6h"]
Oct 31 01:50:16.455241 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:16.455203    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc" path="/var/lib/kubelet/pods/ffc1cc49-0f55-4e0e-9a9e-8419dd417dcc/volumes"
Oct 31 01:50:18.945363 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:18.945325    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:23.645689 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:23.645653    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:28.287089 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:28.287054    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:28.500380 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:28.500347    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:28.697226 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:28.697125    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:29.182263 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:50:29.182226    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:50:33.209114 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:33.209076    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:38.014866 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:38.014830    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:42.189103 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:42.189052    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:43.759714 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:43.759681    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:45.451763 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:45.451729    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:46.188038 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:46.188006    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:46.665276 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:46.665248    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:47.365386 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:47.365353    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:47.633786 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:47.633707    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:49.211447 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:49.211388    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:51.144277 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:51.144243    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:51.351439 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:51.351408    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:51.593383 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:51.593350    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:52.947690 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:52.947658    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:54.713325 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:54.713293    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:54.906164 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:54.906130    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:55.103168 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:55.103136    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:57.464544 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:57.464464    4476 generic.go:356] "Generic (PLEG): container finished" podID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerID="fb6ebf7571a9c70935bbe989502cc41023bd102e899e9361385a4fe02dae4de3" exitCode=0
Oct 31 01:50:57.464544 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:57.464533    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt" event={"ID":"069276a8-4b8e-4db0-9dd1-86c7b402309d","Type":"ContainerDied","Data":"fb6ebf7571a9c70935bbe989502cc41023bd102e899e9361385a4fe02dae4de3"}
Oct 31 01:50:57.465098 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:57.464887    4476 scope.go:122] "RemoveContainer" containerID="fb6ebf7571a9c70935bbe989502cc41023bd102e899e9361385a4fe02dae4de3"
Oct 31 01:50:57.466527 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:57.466496    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:50:57.466660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:57.466582    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:50:57.466660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:57.466630    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:50:58.492109 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:58.492069    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt" event={"ID":"069276a8-4b8e-4db0-9dd1-86c7b402309d","Type":"ContainerStarted","Data":"648774f16c689ca7bbaaf98082af6f1567f58502804d935a18daa9270b6fd896"}
Oct 31 01:50:58.556408 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:58.556382    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:50:58.556676 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:58.556655    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 01:50:58.556737 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:58.556703    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt" podUID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 01:50:58.885366 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:58.885334    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"]
Oct 31 01:50:58.886213 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:58.886169    4476 cpu_manager.go:414] "containerMap: removing container" podUID="a3defef0-a205-49c8-9bb1-ed3b194b329b" containerName="payload-pre-processing"
Oct 31 01:50:58.886213 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:58.886211    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3defef0-a205-49c8-9bb1-ed3b194b329b" containerName="payload-pre-processing"
Oct 31 01:50:58.886395 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:58.886364    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3defef0-a205-49c8-9bb1-ed3b194b329b" containerName="payload-pre-processing"
Oct 31 01:50:58.888631 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:58.888611    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:58.935554 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:58.935520    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"]
Oct 31 01:50:58.941816 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:58.941792    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-41014a\"/\"default-dockercfg-64dvw\""
Oct 31 01:50:58.942090 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:58.942063    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-41014a\"/\"auth-test-model-41014a-kserve-self-signed-certs\""
Oct 31 01:50:58.942235 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:58.942128    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-41014a\"/\"openshift-service-ca.crt\""
Oct 31 01:50:58.942235 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:58.942220    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-41014a\"/\"kube-root-ca.crt\""
Oct 31 01:50:59.011469 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.011426    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:59.024986 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.024960    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-model-cache\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.025379 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.025000    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-dshm\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.025379 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.025091    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xghzn\" (UniqueName: \"kubernetes.io/projected/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-kube-api-access-xghzn\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.025379 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.025144    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-home\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.025379 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.025210    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-tls-certs\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.025379 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.025324    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-tmp-dir\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.126636 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.126607    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-home\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.126842 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.126674    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-tls-certs\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.126842 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.126711    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-tmp-dir\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.126842 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.126771    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-model-cache\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.126842 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.126808    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-dshm\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.127056 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.126871    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xghzn\" (UniqueName: \"kubernetes.io/projected/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-kube-api-access-xghzn\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.127139 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.127114    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-home\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.127256 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.127204    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-tmp-dir\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.127318 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.127250    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-model-cache\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.129246 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.129227    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-dshm\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.129670 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.129633    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-tls-certs\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.187401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.187327    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xghzn\" (UniqueName: \"kubernetes.io/projected/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-kube-api-access-xghzn\") pod \"auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") " pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.201146 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.201120    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:50:59.207349 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:50:59.207321    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:50:59.556650 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.556621    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 01:50:59.557021 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.556667    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt" podUID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 01:50:59.610550 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.610520    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"]
Oct 31 01:50:59.612071 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:50:59.612047    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71e63968_f6f0_45c8_9d7c_d7a222a4c72c.slice/crio-6cfe9823336ec609472698fc4450d6df8cb19eb89c0ae90c85047a6c0171777d WatchSource:0}: Error finding container 6cfe9823336ec609472698fc4450d6df8cb19eb89c0ae90c85047a6c0171777d: Status 404 returned error can't find the container with id 6cfe9823336ec609472698fc4450d6df8cb19eb89c0ae90c85047a6c0171777d
Oct 31 01:50:59.816473 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:50:59.816352    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:00.505026 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:00.504989    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v" event={"ID":"71e63968-f6f0-45c8-9d7c-d7a222a4c72c","Type":"ContainerStarted","Data":"d6b71ab0deb1e11c494467f5055b668d62c17a8b558efa5be62eac2da0c0cede"}
Oct 31 01:51:00.505026 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:00.505031    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v" event={"ID":"71e63968-f6f0-45c8-9d7c-d7a222a4c72c","Type":"ContainerStarted","Data":"6cfe9823336ec609472698fc4450d6df8cb19eb89c0ae90c85047a6c0171777d"}
Oct 31 01:51:00.556781 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:00.556746    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 01:51:00.557134 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:00.556799    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt" podUID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 01:51:00.623642 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:00.623587    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v" podStartSLOduration=2.623572474 podStartE2EDuration="2.623572474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:00.621025848 +0000 UTC m=+3270.310908974" watchObservedRunningTime="2026-10-31 01:51:00.623572474 +0000 UTC m=+3270.313455600"
Oct 31 01:51:01.201651 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:01.201624    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:51:01.557136 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:01.557104    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 01:51:01.557612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:01.557150    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt" podUID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 01:51:02.557077 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:02.557036    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 01:51:02.557284 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:02.557093    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt" podUID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 01:51:03.556829 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:03.556794    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 31 01:51:03.556998 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:03.556845    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt" podUID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 31 01:51:03.684008 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:03.683975    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:04.465593 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:04.465534    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:04.560760 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:04.560730    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:51:04.561161 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:04.561136    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:51:04.561893 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:04.561877    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 01:51:04.699253 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:04.699224    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:08.761058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:08.761028    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:08.978748 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:08.978714    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:09.201254 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:09.201219    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:51:09.213995 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:09.213974    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:51:09.565940 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:09.565911    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:51:09.617146 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:09.617117    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:12.817384 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:12.817289    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:13.016626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:13.016594    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:13.297285 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:13.297251    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:14.934399 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:14.934338    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:19.893766 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:19.893736    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:21.463832 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:21.463802    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:24.820651 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:24.820586    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:26.855875 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:26.855835    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:27.051222 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:27.051167    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:28.674459 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:28.674383    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:29.181788 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:51:29.181756    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:51:29.610297 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:29.610245    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:30.867931 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:30.867908    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:51:30.885141 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:30.885109    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:51:31.228531 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:31.228453    4476 scope.go:122] "RemoveContainer" containerID="571ed9e7c4d2d95bc7dd6e73811fd97f39305635d24e06a3cba4e1bbc2cdb4e2"
Oct 31 01:51:34.344951 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:34.344920    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:39.122302 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:39.122274    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:41.551351 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:41.551312    4476 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal" podUID="7fc072a2ae8e85ec3464a94d2bac4401" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:51:44.100384 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:44.100329    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:48.901326 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:48.901285    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:51.403412 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:51.403383    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:53.766695 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:53.766665    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:57.885401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:57.885354    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:58.360673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:58.360632    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:58.570398 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:51:58.570368    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:59.182149 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:51:59.182114    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:52:01.342647 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:01.342604    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:02.707692 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:02.707660    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:03.346093 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:03.346056    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:08.132496 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:08.132446    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:08.550555 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:08.550524    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:08.751431 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:08.751393    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:12.881042 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:12.881008    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:13.604393 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:13.604359    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:17.574984 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:17.574941    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:17.780358 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:17.780325    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:22.584201 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:22.584140    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:27.402919 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:27.402877    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:29.071653 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:29.071605    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:29.182334 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:52:29.182300    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:52:32.247259 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:32.247157    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:36.997329 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:36.997289    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:38.142341 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:38.142307    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:41.788831 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:41.788561    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:46.544735 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:46.544704    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:51.327445 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:51.327403    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:56.113801 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:52:56.113767    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:59.181837 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:52:59.181794    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:53:00.978313 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:00.978275    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:05.666488 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:05.666456    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:08.527334 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:08.527298    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:10.600890 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:10.600858    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:11.710869 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:11.710833    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"]
Oct 31 01:53:11.711345 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:11.711041    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh" podUID="50c47e81-c94e-4455-88d3-1d4b1c311d75" containerName="istio-proxy" containerID="cri-o://a396658eabc427dfa8e714d631a321fbb650ad5e406ffb5431b4553d20aedb75" gracePeriod=30
Oct 31 01:53:13.940345 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:13.940283    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:15.283438 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:15.283411    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:16.965763 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:16.965738    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:53:17.001150 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.001055    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-workload-certs\") pod \"50c47e81-c94e-4455-88d3-1d4b1c311d75\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") "
Oct 31 01:53:17.001150 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.001100    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-token\" (UniqueName: \"kubernetes.io/projected/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-token\") pod \"50c47e81-c94e-4455-88d3-1d4b1c311d75\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") "
Oct 31 01:53:17.001150 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.001131    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-credential-socket\") pod \"50c47e81-c94e-4455-88d3-1d4b1c311d75\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") "
Oct 31 01:53:17.001150 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.001150    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-podinfo\") pod \"50c47e81-c94e-4455-88d3-1d4b1c311d75\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") "
Oct 31 01:53:17.001551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.001238    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-workload-socket\") pod \"50c47e81-c94e-4455-88d3-1d4b1c311d75\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") "
Oct 31 01:53:17.001551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.001270    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-data\") pod \"50c47e81-c94e-4455-88d3-1d4b1c311d75\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") "
Oct 31 01:53:17.001551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.001324    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c47e81-c94e-4455-88d3-1d4b1c311d75-kube-api-access-qrlsk\" (UniqueName: \"kubernetes.io/projected/50c47e81-c94e-4455-88d3-1d4b1c311d75-kube-api-access-qrlsk\") pod \"50c47e81-c94e-4455-88d3-1d4b1c311d75\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") "
Oct 31 01:53:17.001551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.001398    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-envoy\") pod \"50c47e81-c94e-4455-88d3-1d4b1c311d75\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") "
Oct 31 01:53:17.001551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.001467    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c47e81-c94e-4455-88d3-1d4b1c311d75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c47e81-c94e-4455-88d3-1d4b1c311d75-istiod-ca-cert\") pod \"50c47e81-c94e-4455-88d3-1d4b1c311d75\" (UID: \"50c47e81-c94e-4455-88d3-1d4b1c311d75\") "
Oct 31 01:53:17.001551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.001502    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-workload-certs" pod "50c47e81-c94e-4455-88d3-1d4b1c311d75" (UID: "50c47e81-c94e-4455-88d3-1d4b1c311d75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:17.001551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.001510    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-credential-socket" pod "50c47e81-c94e-4455-88d3-1d4b1c311d75" (UID: "50c47e81-c94e-4455-88d3-1d4b1c311d75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:17.001551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.001532    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-workload-socket" pod "50c47e81-c94e-4455-88d3-1d4b1c311d75" (UID: "50c47e81-c94e-4455-88d3-1d4b1c311d75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:17.002155 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.001741    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-data" pod "50c47e81-c94e-4455-88d3-1d4b1c311d75" (UID: "50c47e81-c94e-4455-88d3-1d4b1c311d75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:17.002155 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.001921    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c47e81-c94e-4455-88d3-1d4b1c311d75-istiod-ca-cert" pod "50c47e81-c94e-4455-88d3-1d4b1c311d75" (UID: "50c47e81-c94e-4455-88d3-1d4b1c311d75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:17.002319 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.002297    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:17.002385 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.002324    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:17.002385 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.002340    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:17.002385 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.002355    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c47e81-c94e-4455-88d3-1d4b1c311d75-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:17.002385 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.002368    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:17.003821 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.003792    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c47e81-c94e-4455-88d3-1d4b1c311d75-kube-api-access-qrlsk" pod "50c47e81-c94e-4455-88d3-1d4b1c311d75" (UID: "50c47e81-c94e-4455-88d3-1d4b1c311d75"). InnerVolumeSpecName "kube-api-access-qrlsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:17.003821 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.003802    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-token" pod "50c47e81-c94e-4455-88d3-1d4b1c311d75" (UID: "50c47e81-c94e-4455-88d3-1d4b1c311d75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:17.003821 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.003793    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-podinfo" pod "50c47e81-c94e-4455-88d3-1d4b1c311d75" (UID: "50c47e81-c94e-4455-88d3-1d4b1c311d75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:53:17.003998 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.003856    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-envoy" pod "50c47e81-c94e-4455-88d3-1d4b1c311d75" (UID: "50c47e81-c94e-4455-88d3-1d4b1c311d75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:17.103073 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.103040    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:17.103073 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.103067    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:17.103073 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.103078    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrlsk\" (UniqueName: \"kubernetes.io/projected/50c47e81-c94e-4455-88d3-1d4b1c311d75-kube-api-access-qrlsk\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:17.103340 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.103088    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c47e81-c94e-4455-88d3-1d4b1c311d75-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:17.328547 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.328512    4476 generic.go:356] "Generic (PLEG): container finished" podID="50c47e81-c94e-4455-88d3-1d4b1c311d75" containerID="a396658eabc427dfa8e714d631a321fbb650ad5e406ffb5431b4553d20aedb75" exitCode=0
Oct 31 01:53:17.328751 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.328561    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"
Oct 31 01:53:17.328751 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.328592    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh" event={"ID":"50c47e81-c94e-4455-88d3-1d4b1c311d75","Type":"ContainerDied","Data":"a396658eabc427dfa8e714d631a321fbb650ad5e406ffb5431b4553d20aedb75"}
Oct 31 01:53:17.328751 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.328648    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh" event={"ID":"50c47e81-c94e-4455-88d3-1d4b1c311d75","Type":"ContainerDied","Data":"e20730a1f63817544ace5c01b1d0b580ca15208681f2abf7b93e249f1481c960"}
Oct 31 01:53:17.328751 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.328670    4476 scope.go:122] "RemoveContainer" containerID="a396658eabc427dfa8e714d631a321fbb650ad5e406ffb5431b4553d20aedb75"
Oct 31 01:53:17.340479 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.340459    4476 scope.go:122] "RemoveContainer" containerID="a396658eabc427dfa8e714d631a321fbb650ad5e406ffb5431b4553d20aedb75"
Oct 31 01:53:17.340766 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:53:17.340748    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a396658eabc427dfa8e714d631a321fbb650ad5e406ffb5431b4553d20aedb75\": container with ID starting with a396658eabc427dfa8e714d631a321fbb650ad5e406ffb5431b4553d20aedb75 not found: ID does not exist" containerID="a396658eabc427dfa8e714d631a321fbb650ad5e406ffb5431b4553d20aedb75"
Oct 31 01:53:17.340818 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.340772    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a396658eabc427dfa8e714d631a321fbb650ad5e406ffb5431b4553d20aedb75"} err="failed to get container status \"a396658eabc427dfa8e714d631a321fbb650ad5e406ffb5431b4553d20aedb75\": rpc error: code = NotFound desc = could not find container \"a396658eabc427dfa8e714d631a321fbb650ad5e406ffb5431b4553d20aedb75\": container with ID starting with a396658eabc427dfa8e714d631a321fbb650ad5e406ffb5431b4553d20aedb75 not found: ID does not exist"
Oct 31 01:53:17.494242 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.494209    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"]
Oct 31 01:53:17.497605 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:17.497578    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-ztrzh"]
Oct 31 01:53:18.454682 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:18.454637    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c47e81-c94e-4455-88d3-1d4b1c311d75" path="/var/lib/kubelet/pods/50c47e81-c94e-4455-88d3-1d4b1c311d75/volumes"
Oct 31 01:53:20.248159 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:20.248129    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:24.916341 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:24.916304    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:28.800102 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:28.800060    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:29.181863 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:53:29.181777    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:53:29.393473 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:29.393442    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:29.706396 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:29.706366    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:30.397449 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:30.397414    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:34.402872 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:34.402836    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:37.319204 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:37.319145    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:39.268749 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:39.268705    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:43.994840 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:43.994788    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:48.929671 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:48.929635    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:53.816788 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:53.816750    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:58.632396 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:53:58.632358    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:59.180963 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:53:59.180930    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:54:03.423524 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:03.423469    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:08.059802 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:08.059751    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:08.251699 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:08.251643    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:12.974541 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:12.974443    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:17.749842 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:17.749803    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:20.472217 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:20.472167    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:22.387821 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:22.387765    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:27.196248 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:27.196210    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:29.180999 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:54:29.180963    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:54:29.716404 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:29.716356    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:29.893045 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:29.893010    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:31.981137 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:31.981096    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:36.672130 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:36.672094    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:41.380740 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:41.380710    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:43.478532 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.478498    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"]
Oct 31 01:54:43.479021 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.479008    4476 cpu_manager.go:414] "containerMap: removing container" podUID="50c47e81-c94e-4455-88d3-1d4b1c311d75" containerName="istio-proxy"
Oct 31 01:54:43.479071 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.479022    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c47e81-c94e-4455-88d3-1d4b1c311d75" containerName="istio-proxy"
Oct 31 01:54:43.479150 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.479141    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c47e81-c94e-4455-88d3-1d4b1c311d75" containerName="istio-proxy"
Oct 31 01:54:43.482549 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.482535    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.586007 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.585976    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-b6f24b\"/\"kube-root-ca.crt\""
Oct 31 01:54:43.586219 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.586142    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-b6f24b\"/\"openshift-service-ca.crt\""
Oct 31 01:54:43.586219 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.586207    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-b6f24b\"/\"default-dockercfg-4m2rn\""
Oct 31 01:54:43.586711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.586698    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-b6f24b\"/\"auth-test-model-b6f24b-kserve-self-signed-certs\""
Oct 31 01:54:43.589509 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.589489    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"]
Oct 31 01:54:43.622483 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.622448    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:43.640445 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.640411    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-dshm\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.640634 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.640464    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-home\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.640709 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.640623    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-model-cache\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.640816 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.640795    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-tmp-dir\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.640946 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.640833    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bec202-d677-4bfa-b29c-0e1de9af7e7f-tls-certs\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.640946 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.640905    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8ll\" (UniqueName: \"kubernetes.io/projected/07bec202-d677-4bfa-b29c-0e1de9af7e7f-kube-api-access-7v8ll\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.742185 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.742079    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-tmp-dir\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.742185 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.742122    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bec202-d677-4bfa-b29c-0e1de9af7e7f-tls-certs\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.742494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.742206    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8ll\" (UniqueName: \"kubernetes.io/projected/07bec202-d677-4bfa-b29c-0e1de9af7e7f-kube-api-access-7v8ll\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.742494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.742310    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-dshm\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.742494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.742345    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-home\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.742494 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.742393    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-model-cache\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.742731 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.742509    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-tmp-dir\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.742823 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.742804    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-model-cache\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.742888 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.742853    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-home\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.744820 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.744790    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-dshm\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.745125 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.745105    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bec202-d677-4bfa-b29c-0e1de9af7e7f-tls-certs\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:43.801127 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:43.801097    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8ll\" (UniqueName: \"kubernetes.io/projected/07bec202-d677-4bfa-b29c-0e1de9af7e7f-kube-api-access-7v8ll\") pod \"auth-test-model-b6f24b-kserve-66444669d9-f22xb\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") " pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:44.094788 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:44.094761    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:44.470401 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:44.470374    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"]
Oct 31 01:54:44.471352 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:54:44.471330    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07bec202_d677_4bfa_b29c_0e1de9af7e7f.slice/crio-7bd483789f4d10dc9995bd17d78e1654102c6eeffb78fd6c06fa5d525d60f3ff WatchSource:0}: Error finding container 7bd483789f4d10dc9995bd17d78e1654102c6eeffb78fd6c06fa5d525d60f3ff: Status 404 returned error can't find the container with id 7bd483789f4d10dc9995bd17d78e1654102c6eeffb78fd6c06fa5d525d60f3ff
Oct 31 01:54:44.852393 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:44.852362    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb" event={"ID":"07bec202-d677-4bfa-b29c-0e1de9af7e7f","Type":"ContainerStarted","Data":"10e6f382110626c05b35284dd287bb8b95bde164a37d24c504c4d4bdcbac62e1"}
Oct 31 01:54:44.852864 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:44.852409    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb" event={"ID":"07bec202-d677-4bfa-b29c-0e1de9af7e7f","Type":"ContainerStarted","Data":"7bd483789f4d10dc9995bd17d78e1654102c6eeffb78fd6c06fa5d525d60f3ff"}
Oct 31 01:54:44.968103 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:44.968049    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb" podStartSLOduration=1.9680370680000001 podStartE2EDuration="1.968037068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:44.966151135 +0000 UTC m=+3494.656034265" watchObservedRunningTime="2026-10-31 01:54:44.968037068 +0000 UTC m=+3494.657920227"
Oct 31 01:54:45.094846 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:45.094816    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:46.134300 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:46.134262    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:50.925742 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:50.925710    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:50.938597 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:50.938569    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"]
Oct 31 01:54:50.944153 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:50.944126    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.004148 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.004117    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"]
Oct 31 01:54:51.049965 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.049940    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c12708\"/\"kube-root-ca.crt\""
Oct 31 01:54:51.050109 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.049972    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c12708\"/\"openshift-service-ca.crt\""
Oct 31 01:54:51.054100 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.054081    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c12708\"/\"default-dockercfg-h9xbt\""
Oct 31 01:54:51.054100 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.054100    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c12708\"/\"explicit-model-c12708-kserve-self-signed-certs\""
Oct 31 01:54:51.126434 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.126392    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61cda870-5a5d-49f1-ab33-e97235eb19e4-tls-certs\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.126434 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.126425    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz9cm\" (UniqueName: \"kubernetes.io/projected/61cda870-5a5d-49f1-ab33-e97235eb19e4-kube-api-access-fz9cm\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.126710 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.126600    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-home\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.126780 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.126720    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-model-cache\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.126854 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.126838    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-tmp-dir\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.126957 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.126939    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-dshm\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.189957 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.189868    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:51.228039 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.228011    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-model-cache\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.228214 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.228091    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-tmp-dir\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.228214 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.228138    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-dshm\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.228214 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.228197    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61cda870-5a5d-49f1-ab33-e97235eb19e4-tls-certs\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.228400 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.228220    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz9cm\" (UniqueName: \"kubernetes.io/projected/61cda870-5a5d-49f1-ab33-e97235eb19e4-kube-api-access-fz9cm\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.228400 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.228296    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-home\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.228514 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.228449    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-model-cache\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.228577 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.228522    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-tmp-dir\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.228672 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.228650    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-home\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.230657 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.230634    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-dshm\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.231047 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.231025    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61cda870-5a5d-49f1-ab33-e97235eb19e4-tls-certs\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.284540 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.284515    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz9cm\" (UniqueName: \"kubernetes.io/projected/61cda870-5a5d-49f1-ab33-e97235eb19e4-kube-api-access-fz9cm\") pod \"explicit-model-c12708-kserve-64fc5958d5-npctc\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") " pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.556793 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.556767    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:51.939270 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:51.939242    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"]
Oct 31 01:54:51.940050 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:54:51.940027    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61cda870_5a5d_49f1_ab33_e97235eb19e4.slice/crio-48a341ce1481018208fcf2bf3922c44f2854e57824e9785caa814a3d8bfc1667 WatchSource:0}: Error finding container 48a341ce1481018208fcf2bf3922c44f2854e57824e9785caa814a3d8bfc1667: Status 404 returned error can't find the container with id 48a341ce1481018208fcf2bf3922c44f2854e57824e9785caa814a3d8bfc1667
Oct 31 01:54:52.899425 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:52.899389    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc" event={"ID":"61cda870-5a5d-49f1-ab33-e97235eb19e4","Type":"ContainerStarted","Data":"f4ff411954ae216bab7d7dc0516ec61b3c1c9ecb3814dee711ba972037aa6885"}
Oct 31 01:54:52.899612 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:52.899437    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc" event={"ID":"61cda870-5a5d-49f1-ab33-e97235eb19e4","Type":"ContainerStarted","Data":"48a341ce1481018208fcf2bf3922c44f2854e57824e9785caa814a3d8bfc1667"}
Oct 31 01:54:53.013568 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:53.013504    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc" podStartSLOduration=3.013490892 podStartE2EDuration="3.013490892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:53.012711421 +0000 UTC m=+3502.702594583" watchObservedRunningTime="2026-10-31 01:54:53.013490892 +0000 UTC m=+3502.703374018"
Oct 31 01:54:53.557553 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:53.557519    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:54:54.094893 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:54.094866    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:54.107516 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:54.107493    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:54.922416 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:54.922388    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 01:54:55.692209 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:54:55.692153    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:59.182107 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:54:59.182055    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:55:00.700315 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:00.700283    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:01.557663 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:01.557633    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:55:01.570594 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:01.570565    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:55:01.969463 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:01.969382    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:55:05.780155 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:05.780124    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:08.301700 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:08.301669    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"]
Oct 31 01:55:08.554141 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:08.554057    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:08.893947 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:08.893837    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:10.824867 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:10.824829    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:15.917934 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:15.917884    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:19.304920 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:19.304886    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:20.803844 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:20.803801    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:21.005043 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:21.005011    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:23.322546 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.322505    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc" podUID="61cda870-5a5d-49f1-ab33-e97235eb19e4" containerName="main" containerID="cri-o://f4ff411954ae216bab7d7dc0516ec61b3c1c9ecb3814dee711ba972037aa6885" gracePeriod=45
Oct 31 01:55:23.595938 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.595911    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:55:23.715582 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.715542    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-dshm\") pod \"61cda870-5a5d-49f1-ab33-e97235eb19e4\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") "
Oct 31 01:55:23.715764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.715607    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61cda870-5a5d-49f1-ab33-e97235eb19e4-kube-api-access-fz9cm\" (UniqueName: \"kubernetes.io/projected/61cda870-5a5d-49f1-ab33-e97235eb19e4-kube-api-access-fz9cm\") pod \"61cda870-5a5d-49f1-ab33-e97235eb19e4\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") "
Oct 31 01:55:23.715764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.715692    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-home\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-home\") pod \"61cda870-5a5d-49f1-ab33-e97235eb19e4\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") "
Oct 31 01:55:23.715919 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.715783    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61cda870-5a5d-49f1-ab33-e97235eb19e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/61cda870-5a5d-49f1-ab33-e97235eb19e4-tls-certs\") pod \"61cda870-5a5d-49f1-ab33-e97235eb19e4\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") "
Oct 31 01:55:23.715919 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.715818    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-model-cache\") pod \"61cda870-5a5d-49f1-ab33-e97235eb19e4\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") "
Oct 31 01:55:23.715919 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.715875    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-tmp-dir\") pod \"61cda870-5a5d-49f1-ab33-e97235eb19e4\" (UID: \"61cda870-5a5d-49f1-ab33-e97235eb19e4\") "
Oct 31 01:55:23.716372 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.716084    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-home" pod "61cda870-5a5d-49f1-ab33-e97235eb19e4" (UID: "61cda870-5a5d-49f1-ab33-e97235eb19e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:23.716499 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.716481    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:23.716713 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.716689    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-tmp-dir" pod "61cda870-5a5d-49f1-ab33-e97235eb19e4" (UID: "61cda870-5a5d-49f1-ab33-e97235eb19e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:23.716997 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.716791    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-model-cache" pod "61cda870-5a5d-49f1-ab33-e97235eb19e4" (UID: "61cda870-5a5d-49f1-ab33-e97235eb19e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:23.719371 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.719318    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61cda870-5a5d-49f1-ab33-e97235eb19e4-tls-certs" pod "61cda870-5a5d-49f1-ab33-e97235eb19e4" (UID: "61cda870-5a5d-49f1-ab33-e97235eb19e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:55:23.719371 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.719342    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61cda870-5a5d-49f1-ab33-e97235eb19e4-kube-api-access-fz9cm" pod "61cda870-5a5d-49f1-ab33-e97235eb19e4" (UID: "61cda870-5a5d-49f1-ab33-e97235eb19e4"). InnerVolumeSpecName "kube-api-access-fz9cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:23.721306 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.721282    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-dshm" pod "61cda870-5a5d-49f1-ab33-e97235eb19e4" (UID: "61cda870-5a5d-49f1-ab33-e97235eb19e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:23.817772 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.817744    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:23.817772 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.817770    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz9cm\" (UniqueName: \"kubernetes.io/projected/61cda870-5a5d-49f1-ab33-e97235eb19e4-kube-api-access-fz9cm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:23.817949 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.817779    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61cda870-5a5d-49f1-ab33-e97235eb19e4-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:23.817949 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.817788    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:23.817949 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:23.817798    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61cda870-5a5d-49f1-ab33-e97235eb19e4-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:24.092518 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:24.092480    4476 generic.go:356] "Generic (PLEG): container finished" podID="61cda870-5a5d-49f1-ab33-e97235eb19e4" containerID="f4ff411954ae216bab7d7dc0516ec61b3c1c9ecb3814dee711ba972037aa6885" exitCode=0
Oct 31 01:55:24.092693 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:24.092534    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"
Oct 31 01:55:24.092693 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:24.092564    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc" event={"ID":"61cda870-5a5d-49f1-ab33-e97235eb19e4","Type":"ContainerDied","Data":"f4ff411954ae216bab7d7dc0516ec61b3c1c9ecb3814dee711ba972037aa6885"}
Oct 31 01:55:24.092693 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:24.092615    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc" event={"ID":"61cda870-5a5d-49f1-ab33-e97235eb19e4","Type":"ContainerDied","Data":"48a341ce1481018208fcf2bf3922c44f2854e57824e9785caa814a3d8bfc1667"}
Oct 31 01:55:24.092693 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:24.092642    4476 scope.go:122] "RemoveContainer" containerID="f4ff411954ae216bab7d7dc0516ec61b3c1c9ecb3814dee711ba972037aa6885"
Oct 31 01:55:24.104152 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:24.104138    4476 scope.go:122] "RemoveContainer" containerID="f4ff411954ae216bab7d7dc0516ec61b3c1c9ecb3814dee711ba972037aa6885"
Oct 31 01:55:24.104410 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:55:24.104391    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ff411954ae216bab7d7dc0516ec61b3c1c9ecb3814dee711ba972037aa6885\": container with ID starting with f4ff411954ae216bab7d7dc0516ec61b3c1c9ecb3814dee711ba972037aa6885 not found: ID does not exist" containerID="f4ff411954ae216bab7d7dc0516ec61b3c1c9ecb3814dee711ba972037aa6885"
Oct 31 01:55:24.104451 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:24.104418    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ff411954ae216bab7d7dc0516ec61b3c1c9ecb3814dee711ba972037aa6885"} err="failed to get container status \"f4ff411954ae216bab7d7dc0516ec61b3c1c9ecb3814dee711ba972037aa6885\": rpc error: code = NotFound desc = could not find container \"f4ff411954ae216bab7d7dc0516ec61b3c1c9ecb3814dee711ba972037aa6885\": container with ID starting with f4ff411954ae216bab7d7dc0516ec61b3c1c9ecb3814dee711ba972037aa6885 not found: ID does not exist"
Oct 31 01:55:24.256135 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:24.256104    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"]
Oct 31 01:55:24.264052 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:24.264022    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c12708/explicit-model-c12708-kserve-64fc5958d5-npctc"]
Oct 31 01:55:24.454714 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:24.454634    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61cda870-5a5d-49f1-ab33-e97235eb19e4" path="/var/lib/kubelet/pods/61cda870-5a5d-49f1-ab33-e97235eb19e4/volumes"
Oct 31 01:55:25.751714 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:25.751676    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:29.181858 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:55:29.181820    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:55:30.057868 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:30.057834    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:30.542136 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:30.542099    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:35.282600 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:35.282564    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:40.111301 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:40.111266    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:44.878236 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:44.878139    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:49.576396 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:49.576360    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:53.565102 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:53.565061    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:55.158975 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:55.158924    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:55.529196 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:55.529155    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:55.717955 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:55.717913    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:56.203125 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:55:56.203080    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:59.181906 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:55:59.181868    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:56:00.389539 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:00.389480    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:02.268497 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:02.268463    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:02.445473 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:02.445443    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:02.657421 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:02.657387    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:03.210767 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:03.210735    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:03.842456 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:03.842409    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:04.155132 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:04.155051    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:05.040786 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:05.040754    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:05.820635 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:05.820598    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:06.016727 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:06.016697    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:06.204685 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:06.204609    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:08.743145 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:08.743117    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:09.760855 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:09.760826    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:12.431885 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:12.431841    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:12.975701 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:12.975670    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"]
Oct 31 01:56:12.975887 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:12.975867    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw" podUID="57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" containerName="istio-proxy" containerID="cri-o://a6c6deb09b78dca2b5c2b9b47095551406eccf5b2f1c04742c9fd2193837083f" gracePeriod=30
Oct 31 01:56:14.484010 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:14.483971    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:16.231794 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:16.231755    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:56:16.232163 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:16.231819    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw" podUID="57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:56:18.244550 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.244487    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:56:18.414589 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.414553    4476 generic.go:356] "Generic (PLEG): container finished" podID="57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" containerID="a6c6deb09b78dca2b5c2b9b47095551406eccf5b2f1c04742c9fd2193837083f" exitCode=0
Oct 31 01:56:18.414793 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.414591    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw" event={"ID":"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6","Type":"ContainerDied","Data":"a6c6deb09b78dca2b5c2b9b47095551406eccf5b2f1c04742c9fd2193837083f"}
Oct 31 01:56:18.414793 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.414616    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"
Oct 31 01:56:18.414793 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.414661    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw" event={"ID":"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6","Type":"ContainerDied","Data":"878c15c8030c9c00a74c2f132a8f6cb3af1b75d25c8879fdb2926954f413379b"}
Oct 31 01:56:18.414793 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.414695    4476 scope.go:122] "RemoveContainer" containerID="a6c6deb09b78dca2b5c2b9b47095551406eccf5b2f1c04742c9fd2193837083f"
Oct 31 01:56:18.423336 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.423303    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-token\" (UniqueName: \"kubernetes.io/projected/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-token\") pod \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") "
Oct 31 01:56:18.423545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.423352    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-workload-certs\") pod \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") "
Oct 31 01:56:18.423545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.423404    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-envoy\") pod \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") "
Oct 31 01:56:18.423545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.423444    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-podinfo\") pod \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") "
Oct 31 01:56:18.423545 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.423528    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-workload-socket\") pod \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") "
Oct 31 01:56:18.423769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.423597    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-credential-socket\") pod \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") "
Oct 31 01:56:18.423769 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.423741    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-kube-api-access-mrc9r\" (UniqueName: \"kubernetes.io/projected/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-kube-api-access-mrc9r\") pod \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") "
Oct 31 01:56:18.423876 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.423794    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istiod-ca-cert\") pod \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") "
Oct 31 01:56:18.423876 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.423819    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-data\") pod \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\" (UID: \"57e9c87c-1fca-4fe9-b2cc-b849f3357ae6\") "
Oct 31 01:56:18.424918 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.424591    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-data" pod "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" (UID: "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:18.425117 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.425056    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-workload-socket" pod "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" (UID: "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:18.425332 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.425305    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-workload-certs" pod "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" (UID: "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:18.425908 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.425884    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istiod-ca-cert" pod "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" (UID: "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:56:18.426011 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.425974    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-credential-socket" pod "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" (UID: "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:18.427057 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.427027    4476 scope.go:122] "RemoveContainer" containerID="a6c6deb09b78dca2b5c2b9b47095551406eccf5b2f1c04742c9fd2193837083f"
Oct 31 01:56:18.427463 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:56:18.427388    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c6deb09b78dca2b5c2b9b47095551406eccf5b2f1c04742c9fd2193837083f\": container with ID starting with a6c6deb09b78dca2b5c2b9b47095551406eccf5b2f1c04742c9fd2193837083f not found: ID does not exist" containerID="a6c6deb09b78dca2b5c2b9b47095551406eccf5b2f1c04742c9fd2193837083f"
Oct 31 01:56:18.427463 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.427425    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c6deb09b78dca2b5c2b9b47095551406eccf5b2f1c04742c9fd2193837083f"} err="failed to get container status \"a6c6deb09b78dca2b5c2b9b47095551406eccf5b2f1c04742c9fd2193837083f\": rpc error: code = NotFound desc = could not find container \"a6c6deb09b78dca2b5c2b9b47095551406eccf5b2f1c04742c9fd2193837083f\": container with ID starting with a6c6deb09b78dca2b5c2b9b47095551406eccf5b2f1c04742c9fd2193837083f not found: ID does not exist"
Oct 31 01:56:18.428156 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.428093    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-token" pod "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" (UID: "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:56:18.428156 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.428142    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-envoy" pod "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" (UID: "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:18.428402 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.428220    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-podinfo" pod "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" (UID: "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:56:18.428557 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.428536    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-kube-api-access-mrc9r" pod "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" (UID: "57e9c87c-1fca-4fe9-b2cc-b849f3357ae6"). InnerVolumeSpecName "kube-api-access-mrc9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:56:18.525582 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.525492    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrc9r\" (UniqueName: \"kubernetes.io/projected/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-kube-api-access-mrc9r\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:18.525582 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.525556    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:18.525582 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.525572    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:18.525839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.525585    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:18.525839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.525598    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:18.525839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.525609    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:18.525839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.525620    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:18.525839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.525633    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:18.525839 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.525647    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:18.829861 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.829813    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"]
Oct 31 01:56:18.878411 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:18.878377    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-ws6tw"]
Oct 31 01:56:19.177457 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:19.177370    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:19.657920 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:19.657890    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:20.454561 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:20.454528    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" path="/var/lib/kubelet/pods/57e9c87c-1fca-4fe9-b2cc-b849f3357ae6/volumes"
Oct 31 01:56:22.283393 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:22.283359    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:23.904135 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:23.904103    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:28.807976 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:28.807937    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:29.181947 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:56:29.181863    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:56:30.399711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:30.399675    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:30.916011 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:30.915980    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:56:30.927073 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:30.927051    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 01:56:33.023541 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:33.023511    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:33.472390 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:33.472359    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:38.269529 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:38.269495    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:42.946490 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:42.946453    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:47.621147 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:47.621097    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:52.246591 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:52.246555    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:56.914377 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:56:56.914337    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:59.182244 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:56:59.182210    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:57:01.657609 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:01.657571    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:06.383664 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:06.383616    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:09.070938 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:09.070875    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:11.075504 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:11.075426    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:13.710100 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:13.710064    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:13.992822 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:13.992734    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:14.165557 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:14.165514    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:15.802935 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:15.802903    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:19.160507 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:19.160463    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:19.995938 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:19.995902    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:20.478515 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:20.478482    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:25.184602 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:25.184559    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:28.882907 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:28.882875    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:29.181139 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:57:29.181059    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:57:29.957415 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:29.957379    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:30.723608 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:30.723572    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:31.716116 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:31.716084    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"]
Oct 31 01:57:31.764807 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:31.764747    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:34.583911 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:34.583869    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:34.789330 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:34.789297    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:39.280434 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:39.280393    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:42.142314 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:42.142282    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:42.321535 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:42.321500    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:43.792156 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:43.792123    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"]
Oct 31 01:57:43.792586 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:43.792349    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2" podUID="79b7fba4-9eff-4f65-a9d5-a7625f98253b" containerName="istio-proxy" containerID="cri-o://ce86eeaa06784847a7fce6b6f34a45e4ec69fc70274fd76aa4ca12ae2313c06b" gracePeriod=30
Oct 31 01:57:43.937362 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:43.937304    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:46.545022 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:46.544988    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:57:46.545500 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:46.545057    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2" podUID="79b7fba4-9eff-4f65-a9d5-a7625f98253b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:57:46.737370 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:46.737329    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v" podUID="71e63968-f6f0-45c8-9d7c-d7a222a4c72c" containerName="main" containerID="cri-o://d6b71ab0deb1e11c494467f5055b668d62c17a8b558efa5be62eac2da0c0cede" gracePeriod=45
Oct 31 01:57:46.954901 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:46.954872    4476 generic.go:356] "Generic (PLEG): container finished" podID="71e63968-f6f0-45c8-9d7c-d7a222a4c72c" containerID="d6b71ab0deb1e11c494467f5055b668d62c17a8b558efa5be62eac2da0c0cede" exitCode=0
Oct 31 01:57:46.955056 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:46.954921    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v" event={"ID":"71e63968-f6f0-45c8-9d7c-d7a222a4c72c","Type":"ContainerDied","Data":"d6b71ab0deb1e11c494467f5055b668d62c17a8b558efa5be62eac2da0c0cede"}
Oct 31 01:57:47.000454 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.000432    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:57:47.070195 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.070131    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-tls-certs\" (UniqueName: \"kubernetes.io/secret/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-tls-certs\") pod \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") "
Oct 31 01:57:47.070195 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.070195    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-dshm\") pod \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") "
Oct 31 01:57:47.070437 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.070241    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-tmp-dir\") pod \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") "
Oct 31 01:57:47.070437 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.070286    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-kube-api-access-xghzn\" (UniqueName: \"kubernetes.io/projected/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-kube-api-access-xghzn\") pod \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") "
Oct 31 01:57:47.070437 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.070323    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-home\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-home\") pod \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") "
Oct 31 01:57:47.070437 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.070349    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-model-cache\") pod \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\" (UID: \"71e63968-f6f0-45c8-9d7c-d7a222a4c72c\") "
Oct 31 01:57:47.070666 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.070533    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-tmp-dir" pod "71e63968-f6f0-45c8-9d7c-d7a222a4c72c" (UID: "71e63968-f6f0-45c8-9d7c-d7a222a4c72c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:47.070728 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.070704    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-model-cache" pod "71e63968-f6f0-45c8-9d7c-d7a222a4c72c" (UID: "71e63968-f6f0-45c8-9d7c-d7a222a4c72c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:47.071034 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.070931    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:47.071034 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.070956    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:47.071034 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.070932    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-home" pod "71e63968-f6f0-45c8-9d7c-d7a222a4c72c" (UID: "71e63968-f6f0-45c8-9d7c-d7a222a4c72c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:47.072652 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.072624    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-kube-api-access-xghzn" pod "71e63968-f6f0-45c8-9d7c-d7a222a4c72c" (UID: "71e63968-f6f0-45c8-9d7c-d7a222a4c72c"). InnerVolumeSpecName "kube-api-access-xghzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:47.072768 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.072624    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-dshm" pod "71e63968-f6f0-45c8-9d7c-d7a222a4c72c" (UID: "71e63968-f6f0-45c8-9d7c-d7a222a4c72c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:47.072768 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.072752    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-tls-certs" pod "71e63968-f6f0-45c8-9d7c-d7a222a4c72c" (UID: "71e63968-f6f0-45c8-9d7c-d7a222a4c72c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:57:47.172496 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.172425    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:47.172496 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.172451    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:47.172496 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.172463    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xghzn\" (UniqueName: \"kubernetes.io/projected/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-kube-api-access-xghzn\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:47.172496 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.172472    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e63968-f6f0-45c8-9d7c-d7a222a4c72c-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:47.961687 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.961655    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"
Oct 31 01:57:47.962116 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.961697    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v" event={"ID":"71e63968-f6f0-45c8-9d7c-d7a222a4c72c","Type":"ContainerDied","Data":"6cfe9823336ec609472698fc4450d6df8cb19eb89c0ae90c85047a6c0171777d"}
Oct 31 01:57:47.962116 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:47.961746    4476 scope.go:122] "RemoveContainer" containerID="d6b71ab0deb1e11c494467f5055b668d62c17a8b558efa5be62eac2da0c0cede"
Oct 31 01:57:48.124220 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:48.124167    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"]
Oct 31 01:57:48.173385 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:48.173354    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-41014a/auth-test-model-41014a-kserve-59cc9f5dd5-jnn5v"]
Oct 31 01:57:48.463696 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:48.463654    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e63968-f6f0-45c8-9d7c-d7a222a4c72c" path="/var/lib/kubelet/pods/71e63968-f6f0-45c8-9d7c-d7a222a4c72c/volumes"
Oct 31 01:57:48.617134 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:48.617090    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:48.971599 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:48.971571    4476 generic.go:356] "Generic (PLEG): container finished" podID="79b7fba4-9eff-4f65-a9d5-a7625f98253b" containerID="ce86eeaa06784847a7fce6b6f34a45e4ec69fc70274fd76aa4ca12ae2313c06b" exitCode=0
Oct 31 01:57:48.971955 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:48.971654    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2" event={"ID":"79b7fba4-9eff-4f65-a9d5-a7625f98253b","Type":"ContainerDied","Data":"ce86eeaa06784847a7fce6b6f34a45e4ec69fc70274fd76aa4ca12ae2313c06b"}
Oct 31 01:57:49.066792 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.066740    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:57:49.095238 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.095209    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-data\") pod \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") "
Oct 31 01:57:49.095372 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.095287    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-credential-socket\") pod \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") "
Oct 31 01:57:49.095372 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.095334    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-workload-socket\") pod \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") "
Oct 31 01:57:49.095456 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.095381    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-workload-certs\") pod \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") "
Oct 31 01:57:49.095456 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.095438    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istiod-ca-cert\") pod \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") "
Oct 31 01:57:49.095551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.095475    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-data" pod "79b7fba4-9eff-4f65-a9d5-a7625f98253b" (UID: "79b7fba4-9eff-4f65-a9d5-a7625f98253b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:49.095551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.095497    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-podinfo\") pod \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") "
Oct 31 01:57:49.095551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.095533    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-credential-socket" pod "79b7fba4-9eff-4f65-a9d5-a7625f98253b" (UID: "79b7fba4-9eff-4f65-a9d5-a7625f98253b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:49.095723 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.095618    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-envoy\") pod \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") "
Oct 31 01:57:49.095723 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.095639    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-workload-socket" pod "79b7fba4-9eff-4f65-a9d5-a7625f98253b" (UID: "79b7fba4-9eff-4f65-a9d5-a7625f98253b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:49.095723 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.095662    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-token\" (UniqueName: \"kubernetes.io/projected/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-token\") pod \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") "
Oct 31 01:57:49.095881 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.095721    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-workload-certs" pod "79b7fba4-9eff-4f65-a9d5-a7625f98253b" (UID: "79b7fba4-9eff-4f65-a9d5-a7625f98253b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:49.095881 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.095755    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b7fba4-9eff-4f65-a9d5-a7625f98253b-kube-api-access-qk6sv\" (UniqueName: \"kubernetes.io/projected/79b7fba4-9eff-4f65-a9d5-a7625f98253b-kube-api-access-qk6sv\") pod \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\" (UID: \"79b7fba4-9eff-4f65-a9d5-a7625f98253b\") "
Oct 31 01:57:49.095968 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.095935    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istiod-ca-cert" pod "79b7fba4-9eff-4f65-a9d5-a7625f98253b" (UID: "79b7fba4-9eff-4f65-a9d5-a7625f98253b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:57:49.096509 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.096483    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.096640 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.096512    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.096640 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.096529    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.096640 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.096544    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.096640 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.096560    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.097925 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.097878    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-envoy" pod "79b7fba4-9eff-4f65-a9d5-a7625f98253b" (UID: "79b7fba4-9eff-4f65-a9d5-a7625f98253b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:49.098043 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.097931    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-podinfo" pod "79b7fba4-9eff-4f65-a9d5-a7625f98253b" (UID: "79b7fba4-9eff-4f65-a9d5-a7625f98253b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:57:49.098043 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.097940    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-token" pod "79b7fba4-9eff-4f65-a9d5-a7625f98253b" (UID: "79b7fba4-9eff-4f65-a9d5-a7625f98253b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:49.098043 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.097969    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b7fba4-9eff-4f65-a9d5-a7625f98253b-kube-api-access-qk6sv" pod "79b7fba4-9eff-4f65-a9d5-a7625f98253b" (UID: "79b7fba4-9eff-4f65-a9d5-a7625f98253b"). InnerVolumeSpecName "kube-api-access-qk6sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:49.197354 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.197319    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk6sv\" (UniqueName: \"kubernetes.io/projected/79b7fba4-9eff-4f65-a9d5-a7625f98253b-kube-api-access-qk6sv\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.197354 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.197349    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.197526 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.197363    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.197526 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.197375    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b7fba4-9eff-4f65-a9d5-a7625f98253b-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.979344 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.979316    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"
Oct 31 01:57:49.979817 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.979341    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2" event={"ID":"79b7fba4-9eff-4f65-a9d5-a7625f98253b","Type":"ContainerDied","Data":"6ae581ea13a0ffa93b7480d197bb503a027d26689f70ccb3adce13236bbeb20b"}
Oct 31 01:57:49.979817 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:49.979411    4476 scope.go:122] "RemoveContainer" containerID="ce86eeaa06784847a7fce6b6f34a45e4ec69fc70274fd76aa4ca12ae2313c06b"
Oct 31 01:57:50.194705 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:50.194672    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"]
Oct 31 01:57:50.244456 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:50.244388    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-sznq2"]
Oct 31 01:57:50.455629 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:50.455599    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b7fba4-9eff-4f65-a9d5-a7625f98253b" path="/var/lib/kubelet/pods/79b7fba4-9eff-4f65-a9d5-a7625f98253b/volumes"
Oct 31 01:57:52.289454 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.289425    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7909-9cb945"]
Oct 31 01:57:52.290024 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.290009    4476 cpu_manager.go:414] "containerMap: removing container" podUID="57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" containerName="istio-proxy"
Oct 31 01:57:52.290069 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.290026    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" containerName="istio-proxy"
Oct 31 01:57:52.290069 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.290038    4476 cpu_manager.go:414] "containerMap: removing container" podUID="71e63968-f6f0-45c8-9d7c-d7a222a4c72c" containerName="main"
Oct 31 01:57:52.290069 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.290043    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e63968-f6f0-45c8-9d7c-d7a222a4c72c" containerName="main"
Oct 31 01:57:52.290069 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.290054    4476 cpu_manager.go:414] "containerMap: removing container" podUID="79b7fba4-9eff-4f65-a9d5-a7625f98253b" containerName="istio-proxy"
Oct 31 01:57:52.290069 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.290060    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b7fba4-9eff-4f65-a9d5-a7625f98253b" containerName="istio-proxy"
Oct 31 01:57:52.290069 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.290069    4476 cpu_manager.go:414] "containerMap: removing container" podUID="61cda870-5a5d-49f1-ab33-e97235eb19e4" containerName="main"
Oct 31 01:57:52.290278 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.290074    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61cda870-5a5d-49f1-ab33-e97235eb19e4" containerName="main"
Oct 31 01:57:52.290278 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.290159    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e9c87c-1fca-4fe9-b2cc-b849f3357ae6" containerName="istio-proxy"
Oct 31 01:57:52.290278 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.290168    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e63968-f6f0-45c8-9d7c-d7a222a4c72c" containerName="main"
Oct 31 01:57:52.290278 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.290187    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b7fba4-9eff-4f65-a9d5-a7625f98253b" containerName="istio-proxy"
Oct 31 01:57:52.290278 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.290194    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="61cda870-5a5d-49f1-ab33-e97235eb19e4" containerName="main"
Oct 31 01:57:52.294765 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.294749    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-9cb945"
Oct 31 01:57:52.345916 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.345883    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7909-9cb945"]
Oct 31 01:57:52.391290 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.391266    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vxhdm\""
Oct 31 01:57:52.433217 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.433166    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4dhf\" (UniqueName: \"kubernetes.io/projected/546c6642-e277-4b96-bc8d-0f25cc170802-kube-api-access-l4dhf\") pod \"test-curl-7909-9cb945\" (UID: \"546c6642-e277-4b96-bc8d-0f25cc170802\") " pod="opendatahub/test-curl-7909-9cb945"
Oct 31 01:57:52.534432 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.534398    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4dhf\" (UniqueName: \"kubernetes.io/projected/546c6642-e277-4b96-bc8d-0f25cc170802-kube-api-access-l4dhf\") pod \"test-curl-7909-9cb945\" (UID: \"546c6642-e277-4b96-bc8d-0f25cc170802\") " pod="opendatahub/test-curl-7909-9cb945"
Oct 31 01:57:52.590542 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.590466    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4dhf\" (UniqueName: \"kubernetes.io/projected/546c6642-e277-4b96-bc8d-0f25cc170802-kube-api-access-l4dhf\") pod \"test-curl-7909-9cb945\" (UID: \"546c6642-e277-4b96-bc8d-0f25cc170802\") " pod="opendatahub/test-curl-7909-9cb945"
Oct 31 01:57:52.608908 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.608880    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-9cb945"
Oct 31 01:57:52.984694 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.984666    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7909-9cb945"]
Oct 31 01:57:52.986435 ip-10-0-139-174 kubenswrapper[4476]: W1031 01:57:52.986400    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod546c6642_e277_4b96_bc8d_0f25cc170802.slice/crio-26a4a061daa2c43005d222f99bafb42b2901a1f55872d463af4878fed0051735 WatchSource:0}: Error finding container 26a4a061daa2c43005d222f99bafb42b2901a1f55872d463af4878fed0051735: Status 404 returned error can't find the container with id 26a4a061daa2c43005d222f99bafb42b2901a1f55872d463af4878fed0051735
Oct 31 01:57:52.988277 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:52.988263    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:57:53.000673 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:53.000639    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7909-9cb945" event={"ID":"546c6642-e277-4b96-bc8d-0f25cc170802","Type":"ContainerStarted","Data":"26a4a061daa2c43005d222f99bafb42b2901a1f55872d463af4878fed0051735"}
Oct 31 01:57:53.251142 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:53.251036    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:55.997333 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:55.997303    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:56.030457 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:56.030412    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7909-9cb945" event={"ID":"546c6642-e277-4b96-bc8d-0f25cc170802","Type":"ContainerStarted","Data":"657f2f1ea95f8b62597dd1df2450720421c83e5b1d8436140d29e0b09e77a03b"}
Oct 31 01:57:56.145460 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:56.145404    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7909-9cb945" podStartSLOduration=1.642896308 podStartE2EDuration="4.145391228s" totalImagesPullingTime="2.50249492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:56.144224273 +0000 UTC m=+3685.834107397" watchObservedRunningTime="2026-10-31 01:57:56.145391228 +0000 UTC m=+3685.835274355"
Oct 31 01:57:57.986681 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:57:57.986649    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:59.182111 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:57:59.182075    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:58:01.176434 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:01.176397    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:01.348195 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:01.348145    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:02.679624 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:02.679589    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:07.331237 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:07.331195    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:09.406880 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:09.406847    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:11.464451 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:11.464411    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:12.005726 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:12.005688    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:12.197770 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:12.197735    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:12.372316 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:12.372282    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:16.499435 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:16.499402    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:16.690860 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:16.690831    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:17.011629 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:17.011594    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:19.661290 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:19.661229    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:19.835168 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:19.835135    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:20.382550 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:20.382498    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:22.072361 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:22.072319    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:25.616090 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:25.616062    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:26.766953 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:26.766918    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:29.182077 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:58:29.182041    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:58:30.416501 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:30.416470    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:31.487282 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:31.487242    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:36.119466 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:36.119434    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:40.735058 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:40.734972    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:45.380770 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:45.380733    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:50.067380 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:50.067351    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:50.492381 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:50.492307    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:51.407268 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:51.407232    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:52.314113 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:52.314084    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:53.888665 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:53.888628    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:54.485996 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:54.485960    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:54.801211 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:54.801155    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:54.999970 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:54.999933    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:57.064945 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:57.064908    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7909-9cb945"]
Oct 31 01:58:57.065449 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:57.065149    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7909-9cb945" podUID="546c6642-e277-4b96-bc8d-0f25cc170802" containerName="test-curl-7909-9cb945" containerID="cri-o://657f2f1ea95f8b62597dd1df2450720421c83e5b1d8436140d29e0b09e77a03b" gracePeriod=2
Oct 31 01:58:57.069134 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:57.069111    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7909-9cb945"]
Oct 31 01:58:57.115763 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:58:57.115727    4476 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7909-9cb945\" is forbidden: User \"system:node:ip-10-0-139-174.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-174.us-west-1.compute.internal' and this object" podUID="546c6642-e277-4b96-bc8d-0f25cc170802" pod="opendatahub/test-curl-7909-9cb945"
Oct 31 01:58:58.672430 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:58.672397    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:58.861844 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:58.861813    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:59.038046 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:59.038013    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:59.184309 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:58:59.184279    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:58:59.387398 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:59.387337    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:59.401416 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:59.401380    4476 generic.go:356] "Generic (PLEG): container finished" podID="546c6642-e277-4b96-bc8d-0f25cc170802" containerID="657f2f1ea95f8b62597dd1df2450720421c83e5b1d8436140d29e0b09e77a03b" exitCode=137
Oct 31 01:58:59.401588 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:59.401455    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26a4a061daa2c43005d222f99bafb42b2901a1f55872d463af4878fed0051735"
Oct 31 01:58:59.405350 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:59.405332    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-9cb945"
Oct 31 01:58:59.510551 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:59.510524    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/546c6642-e277-4b96-bc8d-0f25cc170802-kube-api-access-l4dhf\" (UniqueName: \"kubernetes.io/projected/546c6642-e277-4b96-bc8d-0f25cc170802-kube-api-access-l4dhf\") pod \"546c6642-e277-4b96-bc8d-0f25cc170802\" (UID: \"546c6642-e277-4b96-bc8d-0f25cc170802\") "
Oct 31 01:58:59.512729 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:59.512709    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/546c6642-e277-4b96-bc8d-0f25cc170802-kube-api-access-l4dhf" pod "546c6642-e277-4b96-bc8d-0f25cc170802" (UID: "546c6642-e277-4b96-bc8d-0f25cc170802"). InnerVolumeSpecName "kube-api-access-l4dhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:58:59.612118 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:59.612038    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4dhf\" (UniqueName: \"kubernetes.io/projected/546c6642-e277-4b96-bc8d-0f25cc170802-kube-api-access-l4dhf\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:58:59.620406 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:59.620379    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:59.805941 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:58:59.805908    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:00.406849 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:00.406813    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-9cb945"
Oct 31 01:59:00.453838 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:00.453801    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="546c6642-e277-4b96-bc8d-0f25cc170802" path="/var/lib/kubelet/pods/546c6642-e277-4b96-bc8d-0f25cc170802/volumes"
Oct 31 01:59:00.465222 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:59:00.465192    4476 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7909-9cb945\" is forbidden: User \"system:node:ip-10-0-139-174.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-174.us-west-1.compute.internal' and this object" podUID="546c6642-e277-4b96-bc8d-0f25cc170802" pod="opendatahub/test-curl-7909-9cb945"
Oct 31 01:59:02.864404 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:02.864370    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:03.038830 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:03.038805    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:04.158662 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:04.158637    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:04.398242 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:04.398205    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:04.582239 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:04.582211    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:04.983210 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:04.983104    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:07.578881 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:07.578847    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:07.755032 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:07.755002    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:09.744626 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:09.744583    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:09.923146 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:09.923115    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:10.626569 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:10.626542    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:12.181539 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:12.181507    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:12.778473 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:12.778440    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:12.976790 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:12.976757    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:14.563615 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:14.563584    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:16.865718 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:16.865687    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:17.062367 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:17.062331    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:19.677287 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:19.677256    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:20.036303 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:20.036271    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:20.206834 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:20.206798    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:20.779329 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:20.779296    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:24.308347 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:24.308310    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:24.486660 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:24.486613    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:24.666764 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:24.666672    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:29.144103 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:29.144065    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:29.185051 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:59:29.185014    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 01:59:33.806711 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:33.806661    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:36.082677 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:36.082642    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:38.469966 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:38.469932    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:43.105989 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:43.105931    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:47.891659 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:47.891626    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:52.588450 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:52.588418    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:57.233082 ip-10-0-139-174 kubenswrapper[4476]: I1031 01:59:57.233047    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:59.182017 ip-10-0-139-174 kubenswrapper[4476]: E1031 01:59:59.181977    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:00:00.209707 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.209668    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"]
Oct 31 02:00:00.210263 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.210248    4476 cpu_manager.go:414] "containerMap: removing container" podUID="546c6642-e277-4b96-bc8d-0f25cc170802" containerName="test-curl-7909-9cb945"
Oct 31 02:00:00.210327 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.210264    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="546c6642-e277-4b96-bc8d-0f25cc170802" containerName="test-curl-7909-9cb945"
Oct 31 02:00:00.210381 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.210372    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="546c6642-e277-4b96-bc8d-0f25cc170802" containerName="test-curl-7909-9cb945"
Oct 31 02:00:00.213666 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.213650    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"]
Oct 31 02:00:00.213768 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.213749    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"
Oct 31 02:00:00.217040 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.217021    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"
Oct 31 02:00:00.312800 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.312768    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlr4\" (UniqueName: \"kubernetes.io/projected/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad-kube-api-access-5hlr4\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn\" (UID: \"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"
Oct 31 02:00:00.312988 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.312968    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae1c890-0172-4977-8dd4-c6fb6104c467-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c\" (UID: \"1ae1c890-0172-4977-8dd4-c6fb6104c467\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"
Oct 31 02:00:00.313049 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.313025    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn\" (UID: \"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"
Oct 31 02:00:00.313156 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.313135    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpl2g\" (UniqueName: \"kubernetes.io/projected/1ae1c890-0172-4977-8dd4-c6fb6104c467-kube-api-access-vpl2g\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c\" (UID: \"1ae1c890-0172-4977-8dd4-c6fb6104c467\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"
Oct 31 02:00:00.323941 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.323919    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\""
Oct 31 02:00:00.379575 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.379541    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"]
Oct 31 02:00:00.414020 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.413990    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae1c890-0172-4977-8dd4-c6fb6104c467-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c\" (UID: \"1ae1c890-0172-4977-8dd4-c6fb6104c467\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"
Oct 31 02:00:00.414164 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.414052    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn\" (UID: \"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"
Oct 31 02:00:00.414164 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.414084    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpl2g\" (UniqueName: \"kubernetes.io/projected/1ae1c890-0172-4977-8dd4-c6fb6104c467-kube-api-access-vpl2g\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c\" (UID: \"1ae1c890-0172-4977-8dd4-c6fb6104c467\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"
Oct 31 02:00:00.414303 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.414223    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlr4\" (UniqueName: \"kubernetes.io/projected/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad-kube-api-access-5hlr4\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn\" (UID: \"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"
Oct 31 02:00:00.414873 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.414850    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae1c890-0172-4977-8dd4-c6fb6104c467-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c\" (UID: \"1ae1c890-0172-4977-8dd4-c6fb6104c467\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"
Oct 31 02:00:00.414944 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.414850    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn\" (UID: \"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"
Oct 31 02:00:00.423582 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.423527    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"]
Oct 31 02:00:00.470755 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.470674    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlr4\" (UniqueName: \"kubernetes.io/projected/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad-kube-api-access-5hlr4\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn\" (UID: \"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"
Oct 31 02:00:00.470908 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.470834    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpl2g\" (UniqueName: \"kubernetes.io/projected/1ae1c890-0172-4977-8dd4-c6fb6104c467-kube-api-access-vpl2g\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c\" (UID: \"1ae1c890-0172-4977-8dd4-c6fb6104c467\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"
Oct 31 02:00:00.527420 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.527396    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"
Oct 31 02:00:00.534503 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.534479    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"
Oct 31 02:00:00.906256 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.906228    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"]
Oct 31 02:00:00.908018 ip-10-0-139-174 kubenswrapper[4476]: W1031 02:00:00.907992    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084bdd50_04ae_4c2f_8f1a_b99d08cce4ad.slice/crio-d884b13610b5959bc2aae924f1d9d89860c34aed2f07edc6c44c3444e50e55b8 WatchSource:0}: Error finding container d884b13610b5959bc2aae924f1d9d89860c34aed2f07edc6c44c3444e50e55b8: Status 404 returned error can't find the container with id d884b13610b5959bc2aae924f1d9d89860c34aed2f07edc6c44c3444e50e55b8
Oct 31 02:00:00.957988 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:00.957964    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"]
Oct 31 02:00:00.958967 ip-10-0-139-174 kubenswrapper[4476]: W1031 02:00:00.958943    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae1c890_0172_4977_8dd4_c6fb6104c467.slice/crio-79829c148388448086bfe82dbcaae6375c328ecbb281eda5d2b52e235061c8a5 WatchSource:0}: Error finding container 79829c148388448086bfe82dbcaae6375c328ecbb281eda5d2b52e235061c8a5: Status 404 returned error can't find the container with id 79829c148388448086bfe82dbcaae6375c328ecbb281eda5d2b52e235061c8a5
Oct 31 02:00:01.808468 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:01.808436    4476 generic.go:356] "Generic (PLEG): container finished" podID="1ae1c890-0172-4977-8dd4-c6fb6104c467" containerID="2ed85b21eb590896745cab6fadab49c3f51c4ae5e2bf5c29c58cfdcea4c76153" exitCode=60
Oct 31 02:00:01.808870 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:01.808511    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c" event={"ID":"1ae1c890-0172-4977-8dd4-c6fb6104c467","Type":"ContainerDied","Data":"2ed85b21eb590896745cab6fadab49c3f51c4ae5e2bf5c29c58cfdcea4c76153"}
Oct 31 02:00:01.808870 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:01.808582    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c" event={"ID":"1ae1c890-0172-4977-8dd4-c6fb6104c467","Type":"ContainerStarted","Data":"79829c148388448086bfe82dbcaae6375c328ecbb281eda5d2b52e235061c8a5"}
Oct 31 02:00:01.808870 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:01.808830    4476 scope.go:122] "RemoveContainer" containerID="2ed85b21eb590896745cab6fadab49c3f51c4ae5e2bf5c29c58cfdcea4c76153"
Oct 31 02:00:01.810148 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:01.810124    4476 generic.go:356] "Generic (PLEG): container finished" podID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" containerID="59e1e864e1c9098900377383c869050097ae40d3fc692c860e1d330c55b81ad0" exitCode=60
Oct 31 02:00:01.810262 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:01.810215    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn" event={"ID":"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad","Type":"ContainerDied","Data":"59e1e864e1c9098900377383c869050097ae40d3fc692c860e1d330c55b81ad0"}
Oct 31 02:00:01.810327 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:01.810266    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn" event={"ID":"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad","Type":"ContainerStarted","Data":"d884b13610b5959bc2aae924f1d9d89860c34aed2f07edc6c44c3444e50e55b8"}
Oct 31 02:00:01.810420 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:01.810402    4476 scope.go:122] "RemoveContainer" containerID="59e1e864e1c9098900377383c869050097ae40d3fc692c860e1d330c55b81ad0"
Oct 31 02:00:01.878190 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:01.878150    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:02.818246 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:02.818209    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c" event={"ID":"1ae1c890-0172-4977-8dd4-c6fb6104c467","Type":"ContainerStarted","Data":"02b6914a7fce0a168f02306a7f2f61c18e18c3ce3bf66d28e32b01b1ddab9706"}
Oct 31 02:00:02.820138 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:02.820112    4476 generic.go:356] "Generic (PLEG): container finished" podID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" containerID="4e241f66d497b982b4418dffbaeaccd7569fcbd21f28f3c79e2e2b62a6880155" exitCode=60
Oct 31 02:00:02.820280 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:02.820206    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn" event={"ID":"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad","Type":"ContainerDied","Data":"4e241f66d497b982b4418dffbaeaccd7569fcbd21f28f3c79e2e2b62a6880155"}
Oct 31 02:00:02.820280 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:02.820259    4476 scope.go:122] "RemoveContainer" containerID="59e1e864e1c9098900377383c869050097ae40d3fc692c860e1d330c55b81ad0"
Oct 31 02:00:02.820538 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:02.820524    4476 scope.go:122] "RemoveContainer" containerID="4e241f66d497b982b4418dffbaeaccd7569fcbd21f28f3c79e2e2b62a6880155"
Oct 31 02:00:02.820826 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:00:02.820804    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn_odh-ai-gateway-infra(084bdd50-04ae-4c2f-8f1a-b99d08cce4ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn" podUID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad"
Oct 31 02:00:02.932511 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:02.932435    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c" podStartSLOduration=2.932422147 podStartE2EDuration="2.932422147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:02.931053926 +0000 UTC m=+3812.620937052" watchObservedRunningTime="2026-10-31 02:00:02.932422147 +0000 UTC m=+3812.622305323"
Oct 31 02:00:03.827530 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:03.827497    4476 generic.go:356] "Generic (PLEG): container finished" podID="1ae1c890-0172-4977-8dd4-c6fb6104c467" containerID="02b6914a7fce0a168f02306a7f2f61c18e18c3ce3bf66d28e32b01b1ddab9706" exitCode=60
Oct 31 02:00:03.828009 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:03.827587    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c" event={"ID":"1ae1c890-0172-4977-8dd4-c6fb6104c467","Type":"ContainerDied","Data":"02b6914a7fce0a168f02306a7f2f61c18e18c3ce3bf66d28e32b01b1ddab9706"}
Oct 31 02:00:03.828009 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:03.827656    4476 scope.go:122] "RemoveContainer" containerID="2ed85b21eb590896745cab6fadab49c3f51c4ae5e2bf5c29c58cfdcea4c76153"
Oct 31 02:00:03.828009 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:03.827941    4476 scope.go:122] "RemoveContainer" containerID="02b6914a7fce0a168f02306a7f2f61c18e18c3ce3bf66d28e32b01b1ddab9706"
Oct 31 02:00:03.828267 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:00:03.828242    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c_odh-ai-gateway-infra(1ae1c890-0172-4977-8dd4-c6fb6104c467)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c" podUID="1ae1c890-0172-4977-8dd4-c6fb6104c467"
Oct 31 02:00:03.829577 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:03.829549    4476 scope.go:122] "RemoveContainer" containerID="4e241f66d497b982b4418dffbaeaccd7569fcbd21f28f3c79e2e2b62a6880155"
Oct 31 02:00:03.829831 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:00:03.829814    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn_odh-ai-gateway-infra(084bdd50-04ae-4c2f-8f1a-b99d08cce4ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn" podUID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad"
Oct 31 02:00:04.837142 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:04.837115    4476 scope.go:122] "RemoveContainer" containerID="02b6914a7fce0a168f02306a7f2f61c18e18c3ce3bf66d28e32b01b1ddab9706"
Oct 31 02:00:04.837568 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:00:04.837468    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c_odh-ai-gateway-infra(1ae1c890-0172-4977-8dd4-c6fb6104c467)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c" podUID="1ae1c890-0172-4977-8dd4-c6fb6104c467"
Oct 31 02:00:06.548882 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:06.548850    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:10.105981 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:10.105946    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:10.889977 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:10.889902    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"]
Oct 31 02:00:10.893866 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:10.893851    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:10.925244 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:10.925219    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6c6990-668967998c-76qqj\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:10.925372 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:10.925350    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lntj\" (UniqueName: \"kubernetes.io/projected/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-kube-api-access-5lntj\") pod \"maas-api-e2e-extmod-a-6c6990-668967998c-76qqj\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:10.925413 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:10.925382    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-metrics-certs\") pod \"maas-api-e2e-extmod-a-6c6990-668967998c-76qqj\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:10.925448 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:10.925420    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6c6990-668967998c-76qqj\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:10.996672 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:10.996640    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-6c6990-serving-cert\""
Oct 31 02:00:10.999979 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:10.999958    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"]
Oct 31 02:00:11.026711 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:11.026668    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6c6990-668967998c-76qqj\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:11.026867 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:11.026793    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lntj\" (UniqueName: \"kubernetes.io/projected/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-kube-api-access-5lntj\") pod \"maas-api-e2e-extmod-a-6c6990-668967998c-76qqj\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:11.026867 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:11.026823    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-metrics-certs\") pod \"maas-api-e2e-extmod-a-6c6990-668967998c-76qqj\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:11.026981 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:11.026872    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6c6990-668967998c-76qqj\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:11.027093 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:00:11.027079    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-serving-cert: secret "maas-api-e2e-extmod-a-6c6990-serving-cert" not found
Oct 31 02:00:11.027164 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:00:11.027146    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-maas-api-tls podName:4ea835fa-fab8-4d8b-9b94-7c2478ad41e9 nodeName:}" failed. No retries permitted until 2026-10-31 02:00:11.527127752 +0000 UTC m=+3821.217010864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-maas-api-tls") pod "maas-api-e2e-extmod-a-6c6990-668967998c-76qqj" (UID: "4ea835fa-fab8-4d8b-9b94-7c2478ad41e9") : secret "maas-api-e2e-extmod-a-6c6990-serving-cert" not found
Oct 31 02:00:11.028645 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:11.028616    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6c6990-668967998c-76qqj\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:11.030241 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:11.030220    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-metrics-certs\") pod \"maas-api-e2e-extmod-a-6c6990-668967998c-76qqj\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:11.086292 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:11.086265    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lntj\" (UniqueName: \"kubernetes.io/projected/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-kube-api-access-5lntj\") pod \"maas-api-e2e-extmod-a-6c6990-668967998c-76qqj\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:11.127184 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:11.127150    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:11.304947 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:11.304914    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:11.531971 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:11.531939    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6c6990-668967998c-76qqj\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:11.534576 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:11.534549    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6c6990-668967998c-76qqj\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:11.806690 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:11.806662    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:12.192216 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:12.192169    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"]
Oct 31 02:00:12.193531 ip-10-0-139-174 kubenswrapper[4476]: W1031 02:00:12.193498    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea835fa_fab8_4d8b_9b94_7c2478ad41e9.slice/crio-836df75d356ae55e374b4b4b865e661729b0f0c130945eb76b52d39a3b359f8e WatchSource:0}: Error finding container 836df75d356ae55e374b4b4b865e661729b0f0c130945eb76b52d39a3b359f8e: Status 404 returned error can't find the container with id 836df75d356ae55e374b4b4b865e661729b0f0c130945eb76b52d39a3b359f8e
Oct 31 02:00:12.888263 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:12.888229    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj" event={"ID":"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9","Type":"ContainerStarted","Data":"836df75d356ae55e374b4b4b865e661729b0f0c130945eb76b52d39a3b359f8e"}
Oct 31 02:00:13.897926 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:13.897891    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj" event={"ID":"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9","Type":"ContainerStarted","Data":"68685eba9a9a39a151e0e2bb229ce010febc0727d451d1eed7aa85d2c483bd8b"}
Oct 31 02:00:13.898353 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:13.897951    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:14.012233 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:14.012146    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj" podStartSLOduration=3.191985853 podStartE2EDuration="4.012132973s" totalImagesPullingTime="820.14712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:14.011758859 +0000 UTC m=+3823.701641985" watchObservedRunningTime="2026-10-31 02:00:14.012132973 +0000 UTC m=+3823.702016099"
Oct 31 02:00:15.976454 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:15.976418    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:16.447403 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:16.447374    4476 scope.go:122] "RemoveContainer" containerID="4e241f66d497b982b4418dffbaeaccd7569fcbd21f28f3c79e2e2b62a6880155"
Oct 31 02:00:16.447577 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:16.447490    4476 scope.go:122] "RemoveContainer" containerID="02b6914a7fce0a168f02306a7f2f61c18e18c3ce3bf66d28e32b01b1ddab9706"
Oct 31 02:00:16.918129 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:16.918095    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c" event={"ID":"1ae1c890-0172-4977-8dd4-c6fb6104c467","Type":"ContainerStarted","Data":"74770fc87a1d4472d8e4294097d6f674df93ea18a6c90ae3b19aec62a772185e"}
Oct 31 02:00:16.920027 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:16.919998    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn" event={"ID":"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad","Type":"ContainerStarted","Data":"1035b74844235852f4a1adf452b7a22a23266a6af1be00077256d647173e32ae"}
Oct 31 02:00:17.143742 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:17.143682    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn" podStartSLOduration=17.143669234 podStartE2EDuration="17.143669234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:17.142971465 +0000 UTC m=+3826.832854594" watchObservedRunningTime="2026-10-31 02:00:17.143669234 +0000 UTC m=+3826.833552360"
Oct 31 02:00:17.930775 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:17.930737    4476 generic.go:356] "Generic (PLEG): container finished" podID="1ae1c890-0172-4977-8dd4-c6fb6104c467" containerID="74770fc87a1d4472d8e4294097d6f674df93ea18a6c90ae3b19aec62a772185e" exitCode=60
Oct 31 02:00:17.930940 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:17.930812    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c" event={"ID":"1ae1c890-0172-4977-8dd4-c6fb6104c467","Type":"ContainerDied","Data":"74770fc87a1d4472d8e4294097d6f674df93ea18a6c90ae3b19aec62a772185e"}
Oct 31 02:00:17.930940 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:17.930890    4476 scope.go:122] "RemoveContainer" containerID="02b6914a7fce0a168f02306a7f2f61c18e18c3ce3bf66d28e32b01b1ddab9706"
Oct 31 02:00:17.931376 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:17.931341    4476 scope.go:122] "RemoveContainer" containerID="74770fc87a1d4472d8e4294097d6f674df93ea18a6c90ae3b19aec62a772185e"
Oct 31 02:00:17.931691 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:00:17.931665    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c_odh-ai-gateway-infra(1ae1c890-0172-4977-8dd4-c6fb6104c467)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c" podUID="1ae1c890-0172-4977-8dd4-c6fb6104c467"
Oct 31 02:00:17.933399 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:17.933378    4476 generic.go:356] "Generic (PLEG): container finished" podID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" containerID="1035b74844235852f4a1adf452b7a22a23266a6af1be00077256d647173e32ae" exitCode=60
Oct 31 02:00:17.933506 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:17.933454    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn" event={"ID":"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad","Type":"ContainerDied","Data":"1035b74844235852f4a1adf452b7a22a23266a6af1be00077256d647173e32ae"}
Oct 31 02:00:17.933784 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:17.933768    4476 scope.go:122] "RemoveContainer" containerID="1035b74844235852f4a1adf452b7a22a23266a6af1be00077256d647173e32ae"
Oct 31 02:00:17.934043 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:00:17.934023    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn_odh-ai-gateway-infra(084bdd50-04ae-4c2f-8f1a-b99d08cce4ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn" podUID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad"
Oct 31 02:00:17.944123 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:17.944103    4476 scope.go:122] "RemoveContainer" containerID="4e241f66d497b982b4418dffbaeaccd7569fcbd21f28f3c79e2e2b62a6880155"
Oct 31 02:00:18.042102 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:18.042077    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"]
Oct 31 02:00:18.155910 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:18.155879    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"]
Oct 31 02:00:19.091861 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.091835    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"
Oct 31 02:00:19.124395 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.124375    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"
Oct 31 02:00:19.216158 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.216093    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae1c890-0172-4977-8dd4-c6fb6104c467-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae1c890-0172-4977-8dd4-c6fb6104c467-maas-api-service-ca\") pod \"1ae1c890-0172-4977-8dd4-c6fb6104c467\" (UID: \"1ae1c890-0172-4977-8dd4-c6fb6104c467\") "
Oct 31 02:00:19.216158 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.216131    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae1c890-0172-4977-8dd4-c6fb6104c467-kube-api-access-vpl2g\" (UniqueName: \"kubernetes.io/projected/1ae1c890-0172-4977-8dd4-c6fb6104c467-kube-api-access-vpl2g\") pod \"1ae1c890-0172-4977-8dd4-c6fb6104c467\" (UID: \"1ae1c890-0172-4977-8dd4-c6fb6104c467\") "
Oct 31 02:00:19.216664 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.216215    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad-maas-api-service-ca\") pod \"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad\" (UID: \"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad\") "
Oct 31 02:00:19.216664 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.216361    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad-kube-api-access-5hlr4\" (UniqueName: \"kubernetes.io/projected/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad-kube-api-access-5hlr4\") pod \"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad\" (UID: \"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad\") "
Oct 31 02:00:19.216664 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.216511    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae1c890-0172-4977-8dd4-c6fb6104c467-maas-api-service-ca" pod "1ae1c890-0172-4977-8dd4-c6fb6104c467" (UID: "1ae1c890-0172-4977-8dd4-c6fb6104c467"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:19.216664 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.216563    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad-maas-api-service-ca" pod "084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" (UID: "084bdd50-04ae-4c2f-8f1a-b99d08cce4ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:19.216890 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.216867    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae1c890-0172-4977-8dd4-c6fb6104c467-maas-api-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:19.216890 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.216892    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad-maas-api-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:19.218450 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.218421    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae1c890-0172-4977-8dd4-c6fb6104c467-kube-api-access-vpl2g" pod "1ae1c890-0172-4977-8dd4-c6fb6104c467" (UID: "1ae1c890-0172-4977-8dd4-c6fb6104c467"). InnerVolumeSpecName "kube-api-access-vpl2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:19.218548 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.218452    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad-kube-api-access-5hlr4" pod "084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" (UID: "084bdd50-04ae-4c2f-8f1a-b99d08cce4ad"). InnerVolumeSpecName "kube-api-access-5hlr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:19.317546 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.317515    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hlr4\" (UniqueName: \"kubernetes.io/projected/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad-kube-api-access-5hlr4\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:19.317546 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.317540    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpl2g\" (UniqueName: \"kubernetes.io/projected/1ae1c890-0172-4977-8dd4-c6fb6104c467-kube-api-access-vpl2g\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:19.910593 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.910554    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:00:19.950445 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.950409    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c" event={"ID":"1ae1c890-0172-4977-8dd4-c6fb6104c467","Type":"ContainerDied","Data":"79829c148388448086bfe82dbcaae6375c328ecbb281eda5d2b52e235061c8a5"}
Oct 31 02:00:19.950615 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.950420    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"
Oct 31 02:00:19.950615 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.950471    4476 scope.go:122] "RemoveContainer" containerID="74770fc87a1d4472d8e4294097d6f674df93ea18a6c90ae3b19aec62a772185e"
Oct 31 02:00:19.951828 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.951814    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"
Oct 31 02:00:19.951935 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.951854    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn" event={"ID":"084bdd50-04ae-4c2f-8f1a-b99d08cce4ad","Type":"ContainerDied","Data":"d884b13610b5959bc2aae924f1d9d89860c34aed2f07edc6c44c3444e50e55b8"}
Oct 31 02:00:19.963348 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:19.963314    4476 scope.go:122] "RemoveContainer" containerID="1035b74844235852f4a1adf452b7a22a23266a6af1be00077256d647173e32ae"
Oct 31 02:00:20.144417 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:20.144383    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"]
Oct 31 02:00:20.147157 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:20.147099    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890200-mnfzn"]
Oct 31 02:00:20.257399 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:20.257318    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"]
Oct 31 02:00:20.263648 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:20.263621    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890200-vvr9c"]
Oct 31 02:00:20.462196 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:20.458811    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" path="/var/lib/kubelet/pods/084bdd50-04ae-4c2f-8f1a-b99d08cce4ad/volumes"
Oct 31 02:00:20.469727 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:20.469694    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae1c890-0172-4977-8dd4-c6fb6104c467" path="/var/lib/kubelet/pods/1ae1c890-0172-4977-8dd4-c6fb6104c467/volumes"
Oct 31 02:00:20.626995 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:20.626967    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:20.796748 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:20.796704    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:20.973831 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:20.973752    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:21.146327 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:21.146299    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:21.317117 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:21.317094    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:25.500872 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:25.500836    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:29.180700 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:00:29.180665    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:00:30.168416 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:30.168354    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:34.820130 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:34.820096    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:39.264112 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.264072    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"]
Oct 31 02:00:39.264817 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.264796    4476 cpu_manager.go:414] "containerMap: removing container" podUID="1ae1c890-0172-4977-8dd4-c6fb6104c467" containerName="cleanup"
Oct 31 02:00:39.264928 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.264820    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae1c890-0172-4977-8dd4-c6fb6104c467" containerName="cleanup"
Oct 31 02:00:39.264928 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.264834    4476 cpu_manager.go:414] "containerMap: removing container" podUID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" containerName="cleanup"
Oct 31 02:00:39.264928 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.264841    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" containerName="cleanup"
Oct 31 02:00:39.264928 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.264866    4476 cpu_manager.go:414] "containerMap: removing container" podUID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" containerName="cleanup"
Oct 31 02:00:39.264928 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.264874    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" containerName="cleanup"
Oct 31 02:00:39.264928 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.264897    4476 cpu_manager.go:414] "containerMap: removing container" podUID="1ae1c890-0172-4977-8dd4-c6fb6104c467" containerName="cleanup"
Oct 31 02:00:39.264928 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.264905    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae1c890-0172-4977-8dd4-c6fb6104c467" containerName="cleanup"
Oct 31 02:00:39.264928 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.264924    4476 cpu_manager.go:414] "containerMap: removing container" podUID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" containerName="cleanup"
Oct 31 02:00:39.265397 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.264931    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" containerName="cleanup"
Oct 31 02:00:39.265397 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.265075    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" containerName="cleanup"
Oct 31 02:00:39.265397 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.265094    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae1c890-0172-4977-8dd4-c6fb6104c467" containerName="cleanup"
Oct 31 02:00:39.265397 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.265104    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae1c890-0172-4977-8dd4-c6fb6104c467" containerName="cleanup"
Oct 31 02:00:39.265397 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.265113    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae1c890-0172-4977-8dd4-c6fb6104c467" containerName="cleanup"
Oct 31 02:00:39.265397 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.265124    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" containerName="cleanup"
Oct 31 02:00:39.268521 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.268502    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:39.276839 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.276817    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"]
Oct 31 02:00:39.320316 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.320293    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ce8dbb-serving-cert\""
Oct 31 02:00:39.336977 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.336941    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af72007-50de-4d45-9c18-bdea122f32e6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:39.337121 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.337098    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-metrics-certs\") pod \"maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:39.337196 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.337144    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:39.337321 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.337300    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkkbc\" (UniqueName: \"kubernetes.io/projected/6af72007-50de-4d45-9c18-bdea122f32e6-kube-api-access-fkkbc\") pod \"maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:39.438132 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.438101    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkkbc\" (UniqueName: \"kubernetes.io/projected/6af72007-50de-4d45-9c18-bdea122f32e6-kube-api-access-fkkbc\") pod \"maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:39.438337 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.438261    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af72007-50de-4d45-9c18-bdea122f32e6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:39.438337 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.438331    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-metrics-certs\") pod \"maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:39.438458 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.438364    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:39.438548 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:00:39.438523    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-serving-cert: secret "maas-api-e2e-extmod-a-ce8dbb-serving-cert" not found
Oct 31 02:00:39.438627 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:00:39.438615    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-maas-api-tls podName:6af72007-50de-4d45-9c18-bdea122f32e6 nodeName:}" failed. No retries permitted until 2026-10-31 02:00:39.938594166 +0000 UTC m=+3849.628477292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-maas-api-tls") pod "maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67" (UID: "6af72007-50de-4d45-9c18-bdea122f32e6") : secret "maas-api-e2e-extmod-a-ce8dbb-serving-cert" not found
Oct 31 02:00:39.439010 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.438988    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af72007-50de-4d45-9c18-bdea122f32e6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:39.440883 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.440861    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-metrics-certs\") pod \"maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:39.442437 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.442368    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:39.498193 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.498148    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkkbc\" (UniqueName: \"kubernetes.io/projected/6af72007-50de-4d45-9c18-bdea122f32e6-kube-api-access-fkkbc\") pod \"maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:39.624232 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.624146    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:39.945650 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.945560    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:39.948244 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.948218    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:39.980631 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:39.980605    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:40.183841 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:40.183815    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:40.560014 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:40.559980    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"]
Oct 31 02:00:40.560400 ip-10-0-139-174 kubenswrapper[4476]: W1031 02:00:40.560232    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af72007_50de_4d45_9c18_bdea122f32e6.slice/crio-9a212dcca2441089273fcbe7f7c845260b032400fbb3632ce5f560e6a31a89f4 WatchSource:0}: Error finding container 9a212dcca2441089273fcbe7f7c845260b032400fbb3632ce5f560e6a31a89f4: Status 404 returned error can't find the container with id 9a212dcca2441089273fcbe7f7c845260b032400fbb3632ce5f560e6a31a89f4
Oct 31 02:00:41.089666 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:41.089636    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67" event={"ID":"6af72007-50de-4d45-9c18-bdea122f32e6","Type":"ContainerStarted","Data":"9a212dcca2441089273fcbe7f7c845260b032400fbb3632ce5f560e6a31a89f4"}
Oct 31 02:00:42.097798 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:42.097760    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67" event={"ID":"6af72007-50de-4d45-9c18-bdea122f32e6","Type":"ContainerStarted","Data":"93bebcc190025bfdf79e12d74d5820b7cb709cf42f391b6180e3cc63b8ca616d"}
Oct 31 02:00:42.098317 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:42.097895    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:42.212529 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:42.212483    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67" podStartSLOduration=2.274414863 podStartE2EDuration="3.212468663s" totalImagesPullingTime="938.0538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:42.211028489 +0000 UTC m=+3851.900911607" watchObservedRunningTime="2026-10-31 02:00:42.212468663 +0000 UTC m=+3851.902351789"
Oct 31 02:00:44.299823 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:44.299788    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:48.109810 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:48.109779    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:00:48.967186 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:48.967134    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:53.612616 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:53.612586    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:58.485638 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:00:58.485603    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:59.183638 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:00:59.183593    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:01:03.546827 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:03.546790    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:05.179639 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:05.179559    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:06.486420 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:06.486391    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:07.729886 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:07.729846    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:07.922096 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:07.922060    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:08.374759 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:08.374729    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:09.836108 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:09.836069    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"]
Oct 31 02:01:09.871683 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:09.871652    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:10.420338 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:10.420255    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:13.130857 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:13.130816    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:17.881964 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:17.881929    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:21.458382 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:21.458338    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:21.633768 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:21.633736    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:21.898911 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:21.898881    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:22.838707 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:22.838669    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:24.837509 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:24.837452    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:24.857395 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:24.857359    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb" podUID="07bec202-d677-4bfa-b29c-0e1de9af7e7f" containerName="main" containerID="cri-o://10e6f382110626c05b35284dd287bb8b95bde164a37d24c504c4d4bdcbac62e1" gracePeriod=45
Oct 31 02:01:24.917653 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:24.917594    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb" podUID="07bec202-d677-4bfa-b29c-0e1de9af7e7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": dial tcp 10.133.0.138:8000: connect: connection refused"
Oct 31 02:01:25.014232 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.014199    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:25.127197 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.127153    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 02:01:25.278023 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.277980    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07bec202-d677-4bfa-b29c-0e1de9af7e7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/07bec202-d677-4bfa-b29c-0e1de9af7e7f-tls-certs\") pod \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") "
Oct 31 02:01:25.278023 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.278026    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-dshm\") pod \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") "
Oct 31 02:01:25.278309 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.278101    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-tmp-dir\") pod \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") "
Oct 31 02:01:25.278309 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.278149    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-home\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-home\") pod \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") "
Oct 31 02:01:25.278309 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.278214    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07bec202-d677-4bfa-b29c-0e1de9af7e7f-kube-api-access-7v8ll\" (UniqueName: \"kubernetes.io/projected/07bec202-d677-4bfa-b29c-0e1de9af7e7f-kube-api-access-7v8ll\") pod \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") "
Oct 31 02:01:25.278309 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.278269    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-model-cache\") pod \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\" (UID: \"07bec202-d677-4bfa-b29c-0e1de9af7e7f\") "
Oct 31 02:01:25.279012 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.278967    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-home" pod "07bec202-d677-4bfa-b29c-0e1de9af7e7f" (UID: "07bec202-d677-4bfa-b29c-0e1de9af7e7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:25.279224 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.279191    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-tmp-dir" pod "07bec202-d677-4bfa-b29c-0e1de9af7e7f" (UID: "07bec202-d677-4bfa-b29c-0e1de9af7e7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:25.279421 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.279397    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-model-cache" pod "07bec202-d677-4bfa-b29c-0e1de9af7e7f" (UID: "07bec202-d677-4bfa-b29c-0e1de9af7e7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:25.282103 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.282067    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07bec202-d677-4bfa-b29c-0e1de9af7e7f-tls-certs" pod "07bec202-d677-4bfa-b29c-0e1de9af7e7f" (UID: "07bec202-d677-4bfa-b29c-0e1de9af7e7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:01:25.282230 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.282212    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bec202-d677-4bfa-b29c-0e1de9af7e7f-kube-api-access-7v8ll" pod "07bec202-d677-4bfa-b29c-0e1de9af7e7f" (UID: "07bec202-d677-4bfa-b29c-0e1de9af7e7f"). InnerVolumeSpecName "kube-api-access-7v8ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:25.284089 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.284066    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-dshm" pod "07bec202-d677-4bfa-b29c-0e1de9af7e7f" (UID: "07bec202-d677-4bfa-b29c-0e1de9af7e7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:25.360230 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.360110    4476 generic.go:356] "Generic (PLEG): container finished" podID="07bec202-d677-4bfa-b29c-0e1de9af7e7f" containerID="10e6f382110626c05b35284dd287bb8b95bde164a37d24c504c4d4bdcbac62e1" exitCode=0
Oct 31 02:01:25.360410 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.360229    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb" event={"ID":"07bec202-d677-4bfa-b29c-0e1de9af7e7f","Type":"ContainerDied","Data":"10e6f382110626c05b35284dd287bb8b95bde164a37d24c504c4d4bdcbac62e1"}
Oct 31 02:01:25.360410 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.360267    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb" event={"ID":"07bec202-d677-4bfa-b29c-0e1de9af7e7f","Type":"ContainerDied","Data":"7bd483789f4d10dc9995bd17d78e1654102c6eeffb78fd6c06fa5d525d60f3ff"}
Oct 31 02:01:25.360410 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.360299    4476 scope.go:122] "RemoveContainer" containerID="10e6f382110626c05b35284dd287bb8b95bde164a37d24c504c4d4bdcbac62e1"
Oct 31 02:01:25.363422 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.363388    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"
Oct 31 02:01:25.368482 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.368455    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"]
Oct 31 02:01:25.369459 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.369302    4476 cpu_manager.go:414] "containerMap: removing container" podUID="1ae1c890-0172-4977-8dd4-c6fb6104c467" containerName="cleanup"
Oct 31 02:01:25.369459 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.369323    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae1c890-0172-4977-8dd4-c6fb6104c467" containerName="cleanup"
Oct 31 02:01:25.369459 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.369368    4476 cpu_manager.go:414] "containerMap: removing container" podUID="07bec202-d677-4bfa-b29c-0e1de9af7e7f" containerName="main"
Oct 31 02:01:25.369459 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.369377    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07bec202-d677-4bfa-b29c-0e1de9af7e7f" containerName="main"
Oct 31 02:01:25.369740 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.369533    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="07bec202-d677-4bfa-b29c-0e1de9af7e7f" containerName="main"
Oct 31 02:01:25.369740 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.369547    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="084bdd50-04ae-4c2f-8f1a-b99d08cce4ad" containerName="cleanup"
Oct 31 02:01:25.373457 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.373440    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"
Oct 31 02:01:25.375879 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.375857    4476 scope.go:122] "RemoveContainer" containerID="10e6f382110626c05b35284dd287bb8b95bde164a37d24c504c4d4bdcbac62e1"
Oct 31 02:01:25.376232 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:01:25.376151    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e6f382110626c05b35284dd287bb8b95bde164a37d24c504c4d4bdcbac62e1\": container with ID starting with 10e6f382110626c05b35284dd287bb8b95bde164a37d24c504c4d4bdcbac62e1 not found: ID does not exist" containerID="10e6f382110626c05b35284dd287bb8b95bde164a37d24c504c4d4bdcbac62e1"
Oct 31 02:01:25.376232 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.376196    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e6f382110626c05b35284dd287bb8b95bde164a37d24c504c4d4bdcbac62e1"} err="failed to get container status \"10e6f382110626c05b35284dd287bb8b95bde164a37d24c504c4d4bdcbac62e1\": rpc error: code = NotFound desc = could not find container \"10e6f382110626c05b35284dd287bb8b95bde164a37d24c504c4d4bdcbac62e1\": container with ID starting with 10e6f382110626c05b35284dd287bb8b95bde164a37d24c504c4d4bdcbac62e1 not found: ID does not exist"
Oct 31 02:01:25.379513 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.379493    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:01:25.379579 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.379517    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bec202-d677-4bfa-b29c-0e1de9af7e7f-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:01:25.379579 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.379526    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:01:25.379579 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.379553    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:01:25.379579 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.379565    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07bec202-d677-4bfa-b29c-0e1de9af7e7f-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:01:25.379579 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.379574    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v8ll\" (UniqueName: \"kubernetes.io/projected/07bec202-d677-4bfa-b29c-0e1de9af7e7f-kube-api-access-7v8ll\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:01:25.423457 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.423420    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"]
Oct 31 02:01:25.426696 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.426669    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-fbe06f\""
Oct 31 02:01:25.426848 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.426794    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-fbe06f-dockercfg-qvxrq\""
Oct 31 02:01:25.480768 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.480729    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3199cbdc-b6ba-4212-af9d-fd3d42a6f944-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8\" (UID: \"3199cbdc-b6ba-4212-af9d-fd3d42a6f944\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"
Oct 31 02:01:25.480935 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.480791    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnd9g\" (UniqueName: \"kubernetes.io/projected/3199cbdc-b6ba-4212-af9d-fd3d42a6f944-kube-api-access-cnd9g\") pod \"payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8\" (UID: \"3199cbdc-b6ba-4212-af9d-fd3d42a6f944\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"
Oct 31 02:01:25.527635 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.527599    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"]
Oct 31 02:01:25.534232 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.534206    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b6f24b/auth-test-model-b6f24b-kserve-66444669d9-f22xb"]
Oct 31 02:01:25.581720 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.581672    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3199cbdc-b6ba-4212-af9d-fd3d42a6f944-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8\" (UID: \"3199cbdc-b6ba-4212-af9d-fd3d42a6f944\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"
Oct 31 02:01:25.581907 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.581796    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnd9g\" (UniqueName: \"kubernetes.io/projected/3199cbdc-b6ba-4212-af9d-fd3d42a6f944-kube-api-access-cnd9g\") pod \"payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8\" (UID: \"3199cbdc-b6ba-4212-af9d-fd3d42a6f944\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"
Oct 31 02:01:25.582563 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.582537    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3199cbdc-b6ba-4212-af9d-fd3d42a6f944-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8\" (UID: \"3199cbdc-b6ba-4212-af9d-fd3d42a6f944\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"
Oct 31 02:01:25.638336 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.638266    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnd9g\" (UniqueName: \"kubernetes.io/projected/3199cbdc-b6ba-4212-af9d-fd3d42a6f944-kube-api-access-cnd9g\") pod \"payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8\" (UID: \"3199cbdc-b6ba-4212-af9d-fd3d42a6f944\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"
Oct 31 02:01:25.688476 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:25.688440    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"
Oct 31 02:01:26.068038 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:26.068004    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"]
Oct 31 02:01:26.068696 ip-10-0-139-174 kubenswrapper[4476]: W1031 02:01:26.068669    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3199cbdc_b6ba_4212_af9d_fd3d42a6f944.slice/crio-b5089e67465c96fa96406009ad2ffd162fdfca07e36dbb7ab78b932baf4eec1e WatchSource:0}: Error finding container b5089e67465c96fa96406009ad2ffd162fdfca07e36dbb7ab78b932baf4eec1e: Status 404 returned error can't find the container with id b5089e67465c96fa96406009ad2ffd162fdfca07e36dbb7ab78b932baf4eec1e
Oct 31 02:01:26.367855 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:26.367742    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8" event={"ID":"3199cbdc-b6ba-4212-af9d-fd3d42a6f944","Type":"ContainerStarted","Data":"397c67f9458712e368477562e5d565a31a99dee83a32d9dfab32fa126e59036a"}
Oct 31 02:01:26.367855 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:26.367808    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8" event={"ID":"3199cbdc-b6ba-4212-af9d-fd3d42a6f944","Type":"ContainerStarted","Data":"b5089e67465c96fa96406009ad2ffd162fdfca07e36dbb7ab78b932baf4eec1e"}
Oct 31 02:01:26.368103 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:26.367952    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"
Oct 31 02:01:26.454823 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:26.454785    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07bec202-d677-4bfa-b29c-0e1de9af7e7f" path="/var/lib/kubelet/pods/07bec202-d677-4bfa-b29c-0e1de9af7e7f/volumes"
Oct 31 02:01:26.481111 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:26.481060    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8" podStartSLOduration=1.481047382 podStartE2EDuration="1.481047382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:26.480468576 +0000 UTC m=+3896.170351705" watchObservedRunningTime="2026-10-31 02:01:26.481047382 +0000 UTC m=+3896.170930508"
Oct 31 02:01:27.997908 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:27.997846    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:29.182435 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:01:29.182402    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:01:30.959633 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:30.959605    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 02:01:30.971203 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:30.971153    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 02:01:33.058958 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:33.058892    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:37.377597 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:37.377567    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"
Oct 31 02:01:37.752319 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:37.752208    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:41.115612 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:41.115490    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:41.432726 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:41.432615    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:41.731243 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:41.731127    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:42.012911 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:42.012832    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:42.527222 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:42.527191    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:47.437896 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:47.437862    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:47.608298 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:47.608265    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:48.858429 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:48.858397    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:49.473126 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:49.473091    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:49.808774 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:49.808733    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:52.154250 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:52.154210    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:55.920827 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:55.920788    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:56.126328 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:56.126286    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:56.864344 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:56.864316    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:57.216856 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:57.216759    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:59.181865 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:01:59.181829    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:01:59.940125 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:01:59.940078    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:00.119621 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.119591    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:00.318208 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.318162    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"]
Oct 31 02:02:00.322330 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.322312    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"
Oct 31 02:02:00.433520 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.433490    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"]
Oct 31 02:02:00.434016 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.433993    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-d50516\""
Oct 31 02:02:00.434185 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.434005    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-d50516-dockercfg-9jw8q\""
Oct 31 02:02:00.465316 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.465288    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"]
Oct 31 02:02:00.467695 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.467668    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf7s4\" (UniqueName: \"kubernetes.io/projected/972f6d48-cd71-4ccf-bc10-192462149181-kube-api-access-gf7s4\") pod \"payload-pre-processing-e2e-autores-d50516-74646775f-znft5\" (UID: \"972f6d48-cd71-4ccf-bc10-192462149181\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"
Oct 31 02:02:00.467811 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.467715    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972f6d48-cd71-4ccf-bc10-192462149181-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d50516-74646775f-znft5\" (UID: \"972f6d48-cd71-4ccf-bc10-192462149181\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"
Oct 31 02:02:00.469227 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.469211    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"
Oct 31 02:02:00.524512 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.524477    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"]
Oct 31 02:02:00.569394 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.569299    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839-plugins-config-volume\") pod \"payload-processing-e2e-autores-d50516-bc99c4787-bv9c7\" (UID: \"6d3fd6dd-6237-4bc8-a87d-2ccc8d515839\") " pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"
Oct 31 02:02:00.569598 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.569394    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf7s4\" (UniqueName: \"kubernetes.io/projected/972f6d48-cd71-4ccf-bc10-192462149181-kube-api-access-gf7s4\") pod \"payload-pre-processing-e2e-autores-d50516-74646775f-znft5\" (UID: \"972f6d48-cd71-4ccf-bc10-192462149181\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"
Oct 31 02:02:00.569598 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.569502    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972f6d48-cd71-4ccf-bc10-192462149181-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d50516-74646775f-znft5\" (UID: \"972f6d48-cd71-4ccf-bc10-192462149181\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"
Oct 31 02:02:00.569598 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.569527    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grzsg\" (UniqueName: \"kubernetes.io/projected/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839-kube-api-access-grzsg\") pod \"payload-processing-e2e-autores-d50516-bc99c4787-bv9c7\" (UID: \"6d3fd6dd-6237-4bc8-a87d-2ccc8d515839\") " pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"
Oct 31 02:02:00.570430 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.570405    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972f6d48-cd71-4ccf-bc10-192462149181-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d50516-74646775f-znft5\" (UID: \"972f6d48-cd71-4ccf-bc10-192462149181\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"
Oct 31 02:02:00.625770 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.625737    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf7s4\" (UniqueName: \"kubernetes.io/projected/972f6d48-cd71-4ccf-bc10-192462149181-kube-api-access-gf7s4\") pod \"payload-pre-processing-e2e-autores-d50516-74646775f-znft5\" (UID: \"972f6d48-cd71-4ccf-bc10-192462149181\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"
Oct 31 02:02:00.635543 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.635520    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"
Oct 31 02:02:00.670492 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.670457    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grzsg\" (UniqueName: \"kubernetes.io/projected/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839-kube-api-access-grzsg\") pod \"payload-processing-e2e-autores-d50516-bc99c4787-bv9c7\" (UID: \"6d3fd6dd-6237-4bc8-a87d-2ccc8d515839\") " pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"
Oct 31 02:02:00.670646 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.670598    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839-plugins-config-volume\") pod \"payload-processing-e2e-autores-d50516-bc99c4787-bv9c7\" (UID: \"6d3fd6dd-6237-4bc8-a87d-2ccc8d515839\") " pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"
Oct 31 02:02:00.671354 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.671325    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839-plugins-config-volume\") pod \"payload-processing-e2e-autores-d50516-bc99c4787-bv9c7\" (UID: \"6d3fd6dd-6237-4bc8-a87d-2ccc8d515839\") " pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"
Oct 31 02:02:00.727285 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.727256    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grzsg\" (UniqueName: \"kubernetes.io/projected/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839-kube-api-access-grzsg\") pod \"payload-processing-e2e-autores-d50516-bc99c4787-bv9c7\" (UID: \"6d3fd6dd-6237-4bc8-a87d-2ccc8d515839\") " pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"
Oct 31 02:02:00.782817 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:00.782786    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"
Oct 31 02:02:01.014567 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:01.014539    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"]
Oct 31 02:02:01.015738 ip-10-0-139-174 kubenswrapper[4476]: W1031 02:02:01.015710    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972f6d48_cd71_4ccf_bc10_192462149181.slice/crio-4edc3064f12a0cfe596d123871d49e227621fe0bd6f924860bd3ce3adb8b0969 WatchSource:0}: Error finding container 4edc3064f12a0cfe596d123871d49e227621fe0bd6f924860bd3ce3adb8b0969: Status 404 returned error can't find the container with id 4edc3064f12a0cfe596d123871d49e227621fe0bd6f924860bd3ce3adb8b0969
Oct 31 02:02:01.157018 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:01.156989    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"]
Oct 31 02:02:01.158239 ip-10-0-139-174 kubenswrapper[4476]: W1031 02:02:01.158215    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3fd6dd_6237_4bc8_a87d_2ccc8d515839.slice/crio-e7cd6a0494a557579e97b856bc3b3a1c36790f252fb37ede602262aa2150a2e1 WatchSource:0}: Error finding container e7cd6a0494a557579e97b856bc3b3a1c36790f252fb37ede602262aa2150a2e1: Status 404 returned error can't find the container with id e7cd6a0494a557579e97b856bc3b3a1c36790f252fb37ede602262aa2150a2e1
Oct 31 02:02:01.579688 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:01.579656    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7" event={"ID":"6d3fd6dd-6237-4bc8-a87d-2ccc8d515839","Type":"ContainerStarted","Data":"c7f470690d0809fe2e2acee477f17c278b47bc6ebc4e78800155fa225c7ceca0"}
Oct 31 02:02:01.580198 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:01.579714    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7" event={"ID":"6d3fd6dd-6237-4bc8-a87d-2ccc8d515839","Type":"ContainerStarted","Data":"e7cd6a0494a557579e97b856bc3b3a1c36790f252fb37ede602262aa2150a2e1"}
Oct 31 02:02:01.580198 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:01.579847    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"
Oct 31 02:02:01.581322 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:01.581296    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5" event={"ID":"972f6d48-cd71-4ccf-bc10-192462149181","Type":"ContainerStarted","Data":"2243bda964167939ec187256595fe98ecbd13359172e735881b96f55eebed41d"}
Oct 31 02:02:01.581419 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:01.581338    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5" event={"ID":"972f6d48-cd71-4ccf-bc10-192462149181","Type":"ContainerStarted","Data":"4edc3064f12a0cfe596d123871d49e227621fe0bd6f924860bd3ce3adb8b0969"}
Oct 31 02:02:01.581419 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:01.581359    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"
Oct 31 02:02:01.695758 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:01.695708    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7" podStartSLOduration=1.69569109 podStartE2EDuration="1.69569109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:01.692400934 +0000 UTC m=+3931.382284072" watchObservedRunningTime="2026-10-31 02:02:01.69569109 +0000 UTC m=+3931.385574216"
Oct 31 02:02:01.806718 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:01.806668    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5" podStartSLOduration=1.806654896 podStartE2EDuration="1.806654896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:01.805376563 +0000 UTC m=+3931.495259691" watchObservedRunningTime="2026-10-31 02:02:01.806654896 +0000 UTC m=+3931.496538065"
Oct 31 02:02:02.081051 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:02.081016    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:03.790470 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:03.790437    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:03.980677 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:03.980637    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:06.971307 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:06.971270    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:07.152511 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:07.152477    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:07.397612 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:07.397578    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:08.764929 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:08.764894    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:09.385157 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:09.385128    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:09.746823 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:09.746733    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:10.757249 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:10.757219    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:12.074476 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:12.074442    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:12.590242 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:12.590213    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"
Oct 31 02:02:12.590419 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:12.590277    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"
Oct 31 02:02:15.713279 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:15.713246    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:16.979806 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:16.979772    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:21.686488 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:21.686457    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:21.980470 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:21.980386    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:23.589669 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:23.589636    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:23.779783 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:23.779749    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:23.965141 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:23.965061    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:26.638130 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:26.638098    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:26.820904 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:26.820864    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:27.003343 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:27.003254    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:27.178617 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:27.178583    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:27.669162 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:27.669131    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:28.051068 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:28.051036    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:29.181168 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:02:29.181131    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:02:32.009233 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:32.009203    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:32.203562 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:32.203534    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:32.383349 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:32.383316    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:35.214989 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.214948    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:35.391832 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.391804    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:35.584700 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.584672    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"]
Oct 31 02:02:35.589803 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.589787    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"
Oct 31 02:02:35.643557 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.643531    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"]
Oct 31 02:02:35.664073 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.664046    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c2dm\" (UniqueName: \"kubernetes.io/projected/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14-kube-api-access-4c2dm\") pod \"payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr\" (UID: \"3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"
Oct 31 02:02:35.664277 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.664253    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr\" (UID: \"3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"
Oct 31 02:02:35.689418 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.689389    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-caff19\""
Oct 31 02:02:35.689521 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.689445    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-caff19-dockercfg-zxg8b\""
Oct 31 02:02:35.745865 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.745829    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"]
Oct 31 02:02:35.750566 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.750535    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"
Oct 31 02:02:35.765575 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.765538    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr\" (UID: \"3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"
Oct 31 02:02:35.765728 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.765688    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c2dm\" (UniqueName: \"kubernetes.io/projected/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14-kube-api-access-4c2dm\") pod \"payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr\" (UID: \"3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"
Oct 31 02:02:35.766218 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.766201    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr\" (UID: \"3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"
Oct 31 02:02:35.797591 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.797562    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"]
Oct 31 02:02:35.822896 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.822853    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c2dm\" (UniqueName: \"kubernetes.io/projected/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14-kube-api-access-4c2dm\") pod \"payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr\" (UID: \"3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"
Oct 31 02:02:35.866808 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.866725    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9t9c\" (UniqueName: \"kubernetes.io/projected/dcdec68c-a09e-45ac-8ed8-1aa397ccf660-kube-api-access-k9t9c\") pod \"payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw\" (UID: \"dcdec68c-a09e-45ac-8ed8-1aa397ccf660\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"
Oct 31 02:02:35.866808 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.866762    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcdec68c-a09e-45ac-8ed8-1aa397ccf660-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw\" (UID: \"dcdec68c-a09e-45ac-8ed8-1aa397ccf660\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"
Oct 31 02:02:35.902908 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.902879    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"
Oct 31 02:02:35.975139 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.975097    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9t9c\" (UniqueName: \"kubernetes.io/projected/dcdec68c-a09e-45ac-8ed8-1aa397ccf660-kube-api-access-k9t9c\") pod \"payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw\" (UID: \"dcdec68c-a09e-45ac-8ed8-1aa397ccf660\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"
Oct 31 02:02:35.975280 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.975163    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcdec68c-a09e-45ac-8ed8-1aa397ccf660-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw\" (UID: \"dcdec68c-a09e-45ac-8ed8-1aa397ccf660\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"
Oct 31 02:02:35.976308 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:35.976274    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcdec68c-a09e-45ac-8ed8-1aa397ccf660-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw\" (UID: \"dcdec68c-a09e-45ac-8ed8-1aa397ccf660\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"
Oct 31 02:02:36.032687 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:36.032656    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9t9c\" (UniqueName: \"kubernetes.io/projected/dcdec68c-a09e-45ac-8ed8-1aa397ccf660-kube-api-access-k9t9c\") pod \"payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw\" (UID: \"dcdec68c-a09e-45ac-8ed8-1aa397ccf660\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"
Oct 31 02:02:36.065985 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:36.065957    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"
Oct 31 02:02:36.280798 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:36.280773    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"]
Oct 31 02:02:36.281821 ip-10-0-139-174 kubenswrapper[4476]: W1031 02:02:36.281795    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3a5819_7b4f_4177_9a2b_5cd76e2a7c14.slice/crio-58db966fe6a1af6d2d46bb4ff1c0935ae8d17d2e11b15d409116e93c933395d2 WatchSource:0}: Error finding container 58db966fe6a1af6d2d46bb4ff1c0935ae8d17d2e11b15d409116e93c933395d2: Status 404 returned error can't find the container with id 58db966fe6a1af6d2d46bb4ff1c0935ae8d17d2e11b15d409116e93c933395d2
Oct 31 02:02:36.444522 ip-10-0-139-174 kubenswrapper[4476]: W1031 02:02:36.444487    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdec68c_a09e_45ac_8ed8_1aa397ccf660.slice/crio-807b1d8b2a0558c92e731b9ab0881fcf61a9992cf4d513a21f0fe1512ade69ea WatchSource:0}: Error finding container 807b1d8b2a0558c92e731b9ab0881fcf61a9992cf4d513a21f0fe1512ade69ea: Status 404 returned error can't find the container with id 807b1d8b2a0558c92e731b9ab0881fcf61a9992cf4d513a21f0fe1512ade69ea
Oct 31 02:02:36.445274 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:36.445250    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"]
Oct 31 02:02:36.804052 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:36.804018    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw" event={"ID":"dcdec68c-a09e-45ac-8ed8-1aa397ccf660","Type":"ContainerStarted","Data":"83e87720b3c62cf30962bc65e727b10e1d2067b7c9f96939847ad0155bffafed"}
Oct 31 02:02:36.804264 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:36.804080    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw" event={"ID":"dcdec68c-a09e-45ac-8ed8-1aa397ccf660","Type":"ContainerStarted","Data":"807b1d8b2a0558c92e731b9ab0881fcf61a9992cf4d513a21f0fe1512ade69ea"}
Oct 31 02:02:36.804264 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:36.804209    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"
Oct 31 02:02:36.805733 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:36.805713    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr" event={"ID":"3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14","Type":"ContainerStarted","Data":"94307c68bce61a83c9a45847fe7ed5dd02deb2ea5329f27870b41355d9205c1f"}
Oct 31 02:02:36.805816 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:36.805748    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr" event={"ID":"3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14","Type":"ContainerStarted","Data":"58db966fe6a1af6d2d46bb4ff1c0935ae8d17d2e11b15d409116e93c933395d2"}
Oct 31 02:02:36.805816 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:36.805781    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"
Oct 31 02:02:36.917549 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:36.917498    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw" podStartSLOduration=1.9174809910000001 podStartE2EDuration="1.917480991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:36.915406936 +0000 UTC m=+3966.605290068" watchObservedRunningTime="2026-10-31 02:02:36.917480991 +0000 UTC m=+3966.607364120"
Oct 31 02:02:37.028633 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:37.028590    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr" podStartSLOduration=2.028573598 podStartE2EDuration="2.028573598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:37.027148687 +0000 UTC m=+3966.717031816" watchObservedRunningTime="2026-10-31 02:02:37.028573598 +0000 UTC m=+3966.718456726"
Oct 31 02:02:37.103494 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:37.103430    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:41.769191 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:41.769153    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:42.098562 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:42.098523    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:46.769634 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:46.769548    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:47.814671 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:47.814641    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"
Oct 31 02:02:47.815086 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:47.814714    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"
Oct 31 02:02:51.391853 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:51.391824    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:51.562134 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:51.562102    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:51.734708 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:51.734628    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:56.294400 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:02:56.294368    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:59.182329 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:02:59.182291    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:03:00.974229 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:00.974166    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:05.629120 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:05.629079    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:10.258446 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:10.258414    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:11.094791 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:11.094717    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:14.995736 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:14.995701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:19.813987 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:19.813946    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:22.142707 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:22.142662    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:24.443987 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:24.443927    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:28.792473 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:28.792440    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"]
Oct 31 02:03:28.796823 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:28.796804    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:28.862658 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:28.862621    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-metrics-certs\") pod \"maas-api-e2e-autores-d50516-68c9cfd497-7ww9q\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:28.862841 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:28.862761    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-maas-api-tls\") pod \"maas-api-e2e-autores-d50516-68c9cfd497-7ww9q\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:28.862841 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:28.862789    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj5h6\" (UniqueName: \"kubernetes.io/projected/6b639316-fe8e-40e3-bab9-19c4e66f9796-kube-api-access-rj5h6\") pod \"maas-api-e2e-autores-d50516-68c9cfd497-7ww9q\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:28.862968 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:28.862886    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b639316-fe8e-40e3-bab9-19c4e66f9796-openshift-service-ca\") pod \"maas-api-e2e-autores-d50516-68c9cfd497-7ww9q\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:28.898921 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:28.898900    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-d50516-serving-cert\""
Oct 31 02:03:28.902021 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:28.902002    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"]
Oct 31 02:03:28.963865 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:28.963835    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-metrics-certs\") pod \"maas-api-e2e-autores-d50516-68c9cfd497-7ww9q\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:28.964022 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:28.963920    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-maas-api-tls\") pod \"maas-api-e2e-autores-d50516-68c9cfd497-7ww9q\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:28.964022 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:28.963940    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5h6\" (UniqueName: \"kubernetes.io/projected/6b639316-fe8e-40e3-bab9-19c4e66f9796-kube-api-access-rj5h6\") pod \"maas-api-e2e-autores-d50516-68c9cfd497-7ww9q\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:28.964022 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:28.964011    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b639316-fe8e-40e3-bab9-19c4e66f9796-openshift-service-ca\") pod \"maas-api-e2e-autores-d50516-68c9cfd497-7ww9q\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:28.964136 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:03:28.964055    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-d50516-serving-cert: secret "maas-api-e2e-autores-d50516-serving-cert" not found
Oct 31 02:03:28.964136 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:03:28.964125    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-maas-api-tls podName:6b639316-fe8e-40e3-bab9-19c4e66f9796 nodeName:}" failed. No retries permitted until 2026-10-31 02:03:29.46410486 +0000 UTC m=+4019.153987965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-maas-api-tls") pod "maas-api-e2e-autores-d50516-68c9cfd497-7ww9q" (UID: "6b639316-fe8e-40e3-bab9-19c4e66f9796") : secret "maas-api-e2e-autores-d50516-serving-cert" not found
Oct 31 02:03:28.964643 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:28.964626    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b639316-fe8e-40e3-bab9-19c4e66f9796-openshift-service-ca\") pod \"maas-api-e2e-autores-d50516-68c9cfd497-7ww9q\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:28.966302 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:28.966270    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-metrics-certs\") pod \"maas-api-e2e-autores-d50516-68c9cfd497-7ww9q\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:29.022161 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:29.022134    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:29.023730 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:29.023710    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj5h6\" (UniqueName: \"kubernetes.io/projected/6b639316-fe8e-40e3-bab9-19c4e66f9796-kube-api-access-rj5h6\") pod \"maas-api-e2e-autores-d50516-68c9cfd497-7ww9q\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:29.181313 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:03:29.181215    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:03:29.212484 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:29.212456    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:29.469746 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:29.469671    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-maas-api-tls\") pod \"maas-api-e2e-autores-d50516-68c9cfd497-7ww9q\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:29.472314 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:29.472283    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-maas-api-tls\") pod \"maas-api-e2e-autores-d50516-68c9cfd497-7ww9q\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:29.711386 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:29.711358    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:30.089446 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:30.089420    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"]
Oct 31 02:03:30.090976 ip-10-0-139-174 kubenswrapper[4476]: W1031 02:03:30.090950    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b639316_fe8e_40e3_bab9_19c4e66f9796.slice/crio-5c4c6a7f3a09c1f5d1306f237ea4cd7b1afa90973d2e229fa521fe539e753fb8 WatchSource:0}: Error finding container 5c4c6a7f3a09c1f5d1306f237ea4cd7b1afa90973d2e229fa521fe539e753fb8: Status 404 returned error can't find the container with id 5c4c6a7f3a09c1f5d1306f237ea4cd7b1afa90973d2e229fa521fe539e753fb8
Oct 31 02:03:30.092748 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:30.092731    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 02:03:30.130009 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:30.129978    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q" event={"ID":"6b639316-fe8e-40e3-bab9-19c4e66f9796","Type":"ContainerStarted","Data":"5c4c6a7f3a09c1f5d1306f237ea4cd7b1afa90973d2e229fa521fe539e753fb8"}
Oct 31 02:03:31.137259 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:31.137226    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q" event={"ID":"6b639316-fe8e-40e3-bab9-19c4e66f9796","Type":"ContainerStarted","Data":"c3d79cbfc9a0d969135c23bcebfe9f70b5f0bd3ad36c42b195c7c3a33ca1ff35"}
Oct 31 02:03:31.137665 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:31.137301    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:31.250989 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:31.250901    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q" podStartSLOduration=2.421300998 podStartE2EDuration="3.250887869s" totalImagesPullingTime="829.586871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:03:31.249108745 +0000 UTC m=+4020.938991870" watchObservedRunningTime="2026-10-31 02:03:31.250887869 +0000 UTC m=+4020.940770994"
Oct 31 02:03:33.905755 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:33.905722    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:37.148247 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:37.148219    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:03:38.517606 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:38.517549    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:38.692067 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:38.692035    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:38.871221 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:38.871190    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:42.098659 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:42.098626    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:43.350819 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:43.350790    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:48.071442 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:48.071413    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:52.788691 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:52.788657    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:57.601757 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:03:57.601716    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:59.182048 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:03:59.182010    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:04:02.650426 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:02.650384    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:07.224331 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:07.224298    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:07.403365 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:07.403327    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:07.579258 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:07.579223    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:11.442676 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:11.442643    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:12.217661 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:12.217625    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:16.850186 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:16.850138    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"]
Oct 31 02:04:16.854329 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:16.854312    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:16.911851 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:16.911822    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-caff19-serving-cert\""
Oct 31 02:04:16.914013 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:16.913988    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"]
Oct 31 02:04:17.012961 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.012923    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-metrics-certs\") pod \"maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:17.013138 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.013057    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-maas-api-tls\") pod \"maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:17.013239 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.013219    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26dc7e6-eb1f-4733-930f-836d9703d440-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:17.013306 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.013279    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbmgl\" (UniqueName: \"kubernetes.io/projected/b26dc7e6-eb1f-4733-930f-836d9703d440-kube-api-access-jbmgl\") pod \"maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:17.071704 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.071648    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:17.114891 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.114790    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26dc7e6-eb1f-4733-930f-836d9703d440-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:17.114891 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.114839    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbmgl\" (UniqueName: \"kubernetes.io/projected/b26dc7e6-eb1f-4733-930f-836d9703d440-kube-api-access-jbmgl\") pod \"maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:17.115133 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.114897    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-metrics-certs\") pod \"maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:17.115133 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.115003    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-maas-api-tls\") pod \"maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:17.115279 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:04:17.115219    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-serving-cert: secret "maas-api-e2e-extmod-b-caff19-serving-cert" not found
Oct 31 02:04:17.115336 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:04:17.115287    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-maas-api-tls podName:b26dc7e6-eb1f-4733-930f-836d9703d440 nodeName:}" failed. No retries permitted until 2026-10-31 02:04:17.615265726 +0000 UTC m=+4067.305148835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-maas-api-tls") pod "maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" (UID: "b26dc7e6-eb1f-4733-930f-836d9703d440") : secret "maas-api-e2e-extmod-b-caff19-serving-cert" not found
Oct 31 02:04:17.115656 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.115632    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26dc7e6-eb1f-4733-930f-836d9703d440-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:17.118236 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.118209    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-metrics-certs\") pod \"maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:17.172498 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.172467    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbmgl\" (UniqueName: \"kubernetes.io/projected/b26dc7e6-eb1f-4733-930f-836d9703d440-kube-api-access-jbmgl\") pod \"maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:17.255045 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.255009    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:17.432540 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.432435    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:17.620772 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.620733    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-maas-api-tls\") pod \"maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:17.623363 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.623344    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-maas-api-tls\") pod \"maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:17.767698 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:17.767590    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:18.142749 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:18.142720    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"]
Oct 31 02:04:18.144404 ip-10-0-139-174 kubenswrapper[4476]: W1031 02:04:18.144374    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26dc7e6_eb1f_4733_930f_836d9703d440.slice/crio-f2c3582287d648d2c23ab6056fdee708510564026aa5da22cd0a0e9c8381a510 WatchSource:0}: Error finding container f2c3582287d648d2c23ab6056fdee708510564026aa5da22cd0a0e9c8381a510: Status 404 returned error can't find the container with id f2c3582287d648d2c23ab6056fdee708510564026aa5da22cd0a0e9c8381a510
Oct 31 02:04:18.436598 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:18.436514    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" event={"ID":"b26dc7e6-eb1f-4733-930f-836d9703d440","Type":"ContainerStarted","Data":"f2c3582287d648d2c23ab6056fdee708510564026aa5da22cd0a0e9c8381a510"}
Oct 31 02:04:19.445100 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:19.445013    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" event={"ID":"b26dc7e6-eb1f-4733-930f-836d9703d440","Type":"ContainerStarted","Data":"057fef47551aef63ae2a18cc08982140d083cf7ecd45243861f364be04be5474"}
Oct 31 02:04:19.445531 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:19.445210    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:19.560981 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:19.560926    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" podStartSLOduration=2.654198596 podStartE2EDuration="3.560909241s" totalImagesPullingTime="906.710645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:04:19.558614381 +0000 UTC m=+4069.248497537" watchObservedRunningTime="2026-10-31 02:04:19.560909241 +0000 UTC m=+4069.250792382"
Oct 31 02:04:22.032070 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:22.032036    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:22.481042 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:22.480956    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:25.454497 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:25.454451    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.151:8443/health\": dial tcp 10.133.0.151:8443: connect: connection refused"
Oct 31 02:04:26.786368 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:26.786336    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:29.187631 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:04:29.187597    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:04:29.513228 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:29.513114    4476 generic.go:356] "Generic (PLEG): container finished" podID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerID="057fef47551aef63ae2a18cc08982140d083cf7ecd45243861f364be04be5474" exitCode=1
Oct 31 02:04:29.513228 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:29.513162    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" event={"ID":"b26dc7e6-eb1f-4733-930f-836d9703d440","Type":"ContainerDied","Data":"057fef47551aef63ae2a18cc08982140d083cf7ecd45243861f364be04be5474"}
Oct 31 02:04:29.513610 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:29.513596    4476 scope.go:122] "RemoveContainer" containerID="057fef47551aef63ae2a18cc08982140d083cf7ecd45243861f364be04be5474"
Oct 31 02:04:30.521527 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:30.521487    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" event={"ID":"b26dc7e6-eb1f-4733-930f-836d9703d440","Type":"ContainerStarted","Data":"f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d"}
Oct 31 02:04:30.521960 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:30.521705    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:31.394635 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:31.394601    4476 scope.go:122] "RemoveContainer" containerID="657f2f1ea95f8b62597dd1df2450720421c83e5b1d8436140d29e0b09e77a03b"
Oct 31 02:04:31.846418 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:31.846386    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:36.529638 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:36.529601    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.151:8443/health\": dial tcp 10.133.0.151:8443: connect: connection refused"
Oct 31 02:04:37.087298 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:37.087269    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:37.265143 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:37.265102    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:37.441500 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:37.441418    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:40.582426 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:40.582394    4476 generic.go:356] "Generic (PLEG): container finished" podID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerID="f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d" exitCode=1
Oct 31 02:04:40.582781 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:40.582468    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" event={"ID":"b26dc7e6-eb1f-4733-930f-836d9703d440","Type":"ContainerDied","Data":"f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d"}
Oct 31 02:04:40.582781 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:40.582532    4476 scope.go:122] "RemoveContainer" containerID="057fef47551aef63ae2a18cc08982140d083cf7ecd45243861f364be04be5474"
Oct 31 02:04:40.582901 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:40.582876    4476 scope.go:122] "RemoveContainer" containerID="f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d"
Oct 31 02:04:40.583239 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:04:40.583217    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g_odh-ai-gateway-infra(b26dc7e6-eb1f-4733-930f-836d9703d440)\"" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440"
Oct 31 02:04:41.977454 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:41.977418    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:42.470708 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:42.470674    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:46.913557 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:46.913513    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:47.768252 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:47.768219    4476 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:47.768702 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:47.768685    4476 scope.go:122] "RemoveContainer" containerID="f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d"
Oct 31 02:04:47.769000 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:04:47.768979    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g_odh-ai-gateway-infra(b26dc7e6-eb1f-4733-930f-836d9703d440)\"" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440"
Oct 31 02:04:51.447306 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:51.447269    4476 scope.go:122] "RemoveContainer" containerID="f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d"
Oct 31 02:04:52.144021 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:52.143988    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:52.644068 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:52.644027    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:52.660723 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:52.660690    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" event={"ID":"b26dc7e6-eb1f-4733-930f-836d9703d440","Type":"ContainerStarted","Data":"e88621ec2a69b32dca465dc95436dc9c08d9c70fc6d06366b61bac272033f57b"}
Oct 31 02:04:52.660903 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:52.660889    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:56.873667 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:56.873629    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:58.673021 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:04:58.672994    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:04:59.184351 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:04:59.184312    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:05:01.702163 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:01.702128    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:06.713351 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:06.713300    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:10.323531 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:10.323499    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"]
Oct 31 02:05:10.430688 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:10.430618    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"]
Oct 31 02:05:10.727373 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:10.727269    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:10.900572 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:10.900511    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:11.696593 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:11.696558    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:11.869608 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:11.869553    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:13.541788 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:13.541755    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:15.324341 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.324278    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7" podUID="6d3fd6dd-6237-4bc8-a87d-2ccc8d515839" containerName="payload-processing" containerID="cri-o://c7f470690d0809fe2e2acee477f17c278b47bc6ebc4e78800155fa225c7ceca0" gracePeriod=25
Oct 31 02:05:15.431557 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.431519    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5" podUID="972f6d48-cd71-4ccf-bc10-192462149181" containerName="payload-pre-processing" containerID="cri-o://2243bda964167939ec187256595fe98ecbd13359172e735881b96f55eebed41d" gracePeriod=25
Oct 31 02:05:15.539289 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.539254    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:15.626202 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.625833    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"
Oct 31 02:05:15.708684 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.708652    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:15.733419 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.733398    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"
Oct 31 02:05:15.809152 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.809126    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839-plugins-config-volume\") pod \"6d3fd6dd-6237-4bc8-a87d-2ccc8d515839\" (UID: \"6d3fd6dd-6237-4bc8-a87d-2ccc8d515839\") "
Oct 31 02:05:15.809365 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.809339    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839-kube-api-access-grzsg\" (UniqueName: \"kubernetes.io/projected/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839-kube-api-access-grzsg\") pod \"6d3fd6dd-6237-4bc8-a87d-2ccc8d515839\" (UID: \"6d3fd6dd-6237-4bc8-a87d-2ccc8d515839\") "
Oct 31 02:05:15.809551 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.809520    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839-plugins-config-volume" pod "6d3fd6dd-6237-4bc8-a87d-2ccc8d515839" (UID: "6d3fd6dd-6237-4bc8-a87d-2ccc8d515839"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:15.809776 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.809759    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:15.811752 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.811726    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839-kube-api-access-grzsg" pod "6d3fd6dd-6237-4bc8-a87d-2ccc8d515839" (UID: "6d3fd6dd-6237-4bc8-a87d-2ccc8d515839"). InnerVolumeSpecName "kube-api-access-grzsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:15.813509 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.813482    4476 generic.go:356] "Generic (PLEG): container finished" podID="972f6d48-cd71-4ccf-bc10-192462149181" containerID="2243bda964167939ec187256595fe98ecbd13359172e735881b96f55eebed41d" exitCode=0
Oct 31 02:05:15.813637 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.813570    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"
Oct 31 02:05:15.813637 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.813604    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5" event={"ID":"972f6d48-cd71-4ccf-bc10-192462149181","Type":"ContainerDied","Data":"2243bda964167939ec187256595fe98ecbd13359172e735881b96f55eebed41d"}
Oct 31 02:05:15.813743 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.813672    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5" event={"ID":"972f6d48-cd71-4ccf-bc10-192462149181","Type":"ContainerDied","Data":"4edc3064f12a0cfe596d123871d49e227621fe0bd6f924860bd3ce3adb8b0969"}
Oct 31 02:05:15.813743 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.813707    4476 scope.go:122] "RemoveContainer" containerID="2243bda964167939ec187256595fe98ecbd13359172e735881b96f55eebed41d"
Oct 31 02:05:15.815143 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.815102    4476 generic.go:356] "Generic (PLEG): container finished" podID="6d3fd6dd-6237-4bc8-a87d-2ccc8d515839" containerID="c7f470690d0809fe2e2acee477f17c278b47bc6ebc4e78800155fa225c7ceca0" exitCode=0
Oct 31 02:05:15.815143 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.815142    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7" event={"ID":"6d3fd6dd-6237-4bc8-a87d-2ccc8d515839","Type":"ContainerDied","Data":"c7f470690d0809fe2e2acee477f17c278b47bc6ebc4e78800155fa225c7ceca0"}
Oct 31 02:05:15.815143 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.815151    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"
Oct 31 02:05:15.815377 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.815169    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7" event={"ID":"6d3fd6dd-6237-4bc8-a87d-2ccc8d515839","Type":"ContainerDied","Data":"e7cd6a0494a557579e97b856bc3b3a1c36790f252fb37ede602262aa2150a2e1"}
Oct 31 02:05:15.829700 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.829623    4476 scope.go:122] "RemoveContainer" containerID="2243bda964167939ec187256595fe98ecbd13359172e735881b96f55eebed41d"
Oct 31 02:05:15.829959 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:05:15.829940    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2243bda964167939ec187256595fe98ecbd13359172e735881b96f55eebed41d\": container with ID starting with 2243bda964167939ec187256595fe98ecbd13359172e735881b96f55eebed41d not found: ID does not exist" containerID="2243bda964167939ec187256595fe98ecbd13359172e735881b96f55eebed41d"
Oct 31 02:05:15.830041 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.829964    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2243bda964167939ec187256595fe98ecbd13359172e735881b96f55eebed41d"} err="failed to get container status \"2243bda964167939ec187256595fe98ecbd13359172e735881b96f55eebed41d\": rpc error: code = NotFound desc = could not find container \"2243bda964167939ec187256595fe98ecbd13359172e735881b96f55eebed41d\": container with ID starting with 2243bda964167939ec187256595fe98ecbd13359172e735881b96f55eebed41d not found: ID does not exist"
Oct 31 02:05:15.830041 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.829985    4476 scope.go:122] "RemoveContainer" containerID="c7f470690d0809fe2e2acee477f17c278b47bc6ebc4e78800155fa225c7ceca0"
Oct 31 02:05:15.841076 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.841062    4476 scope.go:122] "RemoveContainer" containerID="c7f470690d0809fe2e2acee477f17c278b47bc6ebc4e78800155fa225c7ceca0"
Oct 31 02:05:15.841372 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:05:15.841352    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f470690d0809fe2e2acee477f17c278b47bc6ebc4e78800155fa225c7ceca0\": container with ID starting with c7f470690d0809fe2e2acee477f17c278b47bc6ebc4e78800155fa225c7ceca0 not found: ID does not exist" containerID="c7f470690d0809fe2e2acee477f17c278b47bc6ebc4e78800155fa225c7ceca0"
Oct 31 02:05:15.841446 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.841376    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f470690d0809fe2e2acee477f17c278b47bc6ebc4e78800155fa225c7ceca0"} err="failed to get container status \"c7f470690d0809fe2e2acee477f17c278b47bc6ebc4e78800155fa225c7ceca0\": rpc error: code = NotFound desc = could not find container \"c7f470690d0809fe2e2acee477f17c278b47bc6ebc4e78800155fa225c7ceca0\": container with ID starting with c7f470690d0809fe2e2acee477f17c278b47bc6ebc4e78800155fa225c7ceca0 not found: ID does not exist"
Oct 31 02:05:15.911272 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.911237    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972f6d48-cd71-4ccf-bc10-192462149181-kube-api-access-gf7s4\" (UniqueName: \"kubernetes.io/projected/972f6d48-cd71-4ccf-bc10-192462149181-kube-api-access-gf7s4\") pod \"972f6d48-cd71-4ccf-bc10-192462149181\" (UID: \"972f6d48-cd71-4ccf-bc10-192462149181\") "
Oct 31 02:05:15.911448 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.911348    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/972f6d48-cd71-4ccf-bc10-192462149181-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972f6d48-cd71-4ccf-bc10-192462149181-plugins-config-volume\") pod \"972f6d48-cd71-4ccf-bc10-192462149181\" (UID: \"972f6d48-cd71-4ccf-bc10-192462149181\") "
Oct 31 02:05:15.911817 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.911788    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972f6d48-cd71-4ccf-bc10-192462149181-plugins-config-volume" pod "972f6d48-cd71-4ccf-bc10-192462149181" (UID: "972f6d48-cd71-4ccf-bc10-192462149181"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:15.911988 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.911936    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grzsg\" (UniqueName: \"kubernetes.io/projected/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839-kube-api-access-grzsg\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:15.911988 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.911965    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972f6d48-cd71-4ccf-bc10-192462149181-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:15.913581 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:15.913556    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972f6d48-cd71-4ccf-bc10-192462149181-kube-api-access-gf7s4" pod "972f6d48-cd71-4ccf-bc10-192462149181" (UID: "972f6d48-cd71-4ccf-bc10-192462149181"). InnerVolumeSpecName "kube-api-access-gf7s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:16.012884 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:16.012849    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf7s4\" (UniqueName: \"kubernetes.io/projected/972f6d48-cd71-4ccf-bc10-192462149181-kube-api-access-gf7s4\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:16.063165 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:16.063128    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"]
Oct 31 02:05:16.065849 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:16.065828    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d50516-bc99c4787-bv9c7"]
Oct 31 02:05:16.233444 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:16.233405    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"]
Oct 31 02:05:16.236615 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:16.236591    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d50516-74646775f-znft5"]
Oct 31 02:05:16.454856 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:16.454789    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3fd6dd-6237-4bc8-a87d-2ccc8d515839" path="/var/lib/kubelet/pods/6d3fd6dd-6237-4bc8-a87d-2ccc8d515839/volumes"
Oct 31 02:05:16.455214 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:16.455205    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="972f6d48-cd71-4ccf-bc10-192462149181" path="/var/lib/kubelet/pods/972f6d48-cd71-4ccf-bc10-192462149181/volumes"
Oct 31 02:05:16.579496 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:16.579462    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:20.509898 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.509857    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"]
Oct 31 02:05:20.510361 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.510094    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q" podUID="6b639316-fe8e-40e3-bab9-19c4e66f9796" containerName="maas-api" containerID="cri-o://c3d79cbfc9a0d969135c23bcebfe9f70b5f0bd3ad36c42b195c7c3a33ca1ff35" gracePeriod=30
Oct 31 02:05:20.652486 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.652431    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:20.776198 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.776119    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:05:20.849290 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.849257    4476 generic.go:356] "Generic (PLEG): container finished" podID="6b639316-fe8e-40e3-bab9-19c4e66f9796" containerID="c3d79cbfc9a0d969135c23bcebfe9f70b5f0bd3ad36c42b195c7c3a33ca1ff35" exitCode=0
Oct 31 02:05:20.849443 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.849307    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"
Oct 31 02:05:20.849443 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.849304    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q" event={"ID":"6b639316-fe8e-40e3-bab9-19c4e66f9796","Type":"ContainerDied","Data":"c3d79cbfc9a0d969135c23bcebfe9f70b5f0bd3ad36c42b195c7c3a33ca1ff35"}
Oct 31 02:05:20.849443 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.849431    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q" event={"ID":"6b639316-fe8e-40e3-bab9-19c4e66f9796","Type":"ContainerDied","Data":"5c4c6a7f3a09c1f5d1306f237ea4cd7b1afa90973d2e229fa521fe539e753fb8"}
Oct 31 02:05:20.849643 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.849457    4476 scope.go:122] "RemoveContainer" containerID="c3d79cbfc9a0d969135c23bcebfe9f70b5f0bd3ad36c42b195c7c3a33ca1ff35"
Oct 31 02:05:20.859095 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.859077    4476 scope.go:122] "RemoveContainer" containerID="c3d79cbfc9a0d969135c23bcebfe9f70b5f0bd3ad36c42b195c7c3a33ca1ff35"
Oct 31 02:05:20.859404 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:05:20.859384    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d79cbfc9a0d969135c23bcebfe9f70b5f0bd3ad36c42b195c7c3a33ca1ff35\": container with ID starting with c3d79cbfc9a0d969135c23bcebfe9f70b5f0bd3ad36c42b195c7c3a33ca1ff35 not found: ID does not exist" containerID="c3d79cbfc9a0d969135c23bcebfe9f70b5f0bd3ad36c42b195c7c3a33ca1ff35"
Oct 31 02:05:20.859493 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.859411    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d79cbfc9a0d969135c23bcebfe9f70b5f0bd3ad36c42b195c7c3a33ca1ff35"} err="failed to get container status \"c3d79cbfc9a0d969135c23bcebfe9f70b5f0bd3ad36c42b195c7c3a33ca1ff35\": rpc error: code = NotFound desc = could not find container \"c3d79cbfc9a0d969135c23bcebfe9f70b5f0bd3ad36c42b195c7c3a33ca1ff35\": container with ID starting with c3d79cbfc9a0d969135c23bcebfe9f70b5f0bd3ad36c42b195c7c3a33ca1ff35 not found: ID does not exist"
Oct 31 02:05:20.867817 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.867798    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-metrics-certs\") pod \"6b639316-fe8e-40e3-bab9-19c4e66f9796\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") "
Oct 31 02:05:20.867927 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.867914    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b639316-fe8e-40e3-bab9-19c4e66f9796-kube-api-access-rj5h6\" (UniqueName: \"kubernetes.io/projected/6b639316-fe8e-40e3-bab9-19c4e66f9796-kube-api-access-rj5h6\") pod \"6b639316-fe8e-40e3-bab9-19c4e66f9796\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") "
Oct 31 02:05:20.868018 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.868003    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-maas-api-tls\") pod \"6b639316-fe8e-40e3-bab9-19c4e66f9796\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") "
Oct 31 02:05:20.868068 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.868053    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b639316-fe8e-40e3-bab9-19c4e66f9796-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b639316-fe8e-40e3-bab9-19c4e66f9796-openshift-service-ca\") pod \"6b639316-fe8e-40e3-bab9-19c4e66f9796\" (UID: \"6b639316-fe8e-40e3-bab9-19c4e66f9796\") "
Oct 31 02:05:20.868497 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.868470    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b639316-fe8e-40e3-bab9-19c4e66f9796-openshift-service-ca" pod "6b639316-fe8e-40e3-bab9-19c4e66f9796" (UID: "6b639316-fe8e-40e3-bab9-19c4e66f9796"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:20.870091 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.870066    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b639316-fe8e-40e3-bab9-19c4e66f9796-kube-api-access-rj5h6" pod "6b639316-fe8e-40e3-bab9-19c4e66f9796" (UID: "6b639316-fe8e-40e3-bab9-19c4e66f9796"). InnerVolumeSpecName "kube-api-access-rj5h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:20.870207 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.870092    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-metrics-certs" pod "6b639316-fe8e-40e3-bab9-19c4e66f9796" (UID: "6b639316-fe8e-40e3-bab9-19c4e66f9796"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:05:20.870207 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.870119    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-maas-api-tls" pod "6b639316-fe8e-40e3-bab9-19c4e66f9796" (UID: "6b639316-fe8e-40e3-bab9-19c4e66f9796"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:05:20.970112 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.970080    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:20.970112 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.970109    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b639316-fe8e-40e3-bab9-19c4e66f9796-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:20.970342 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.970125    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b639316-fe8e-40e3-bab9-19c4e66f9796-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:20.970342 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:20.970135    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj5h6\" (UniqueName: \"kubernetes.io/projected/6b639316-fe8e-40e3-bab9-19c4e66f9796-kube-api-access-rj5h6\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:21.271910 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:21.271878    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"]
Oct 31 02:05:21.275807 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:21.275780    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d50516-68c9cfd497-7ww9q"]
Oct 31 02:05:22.127189 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:22.127139    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:22.327745 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:22.327700    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:22.454809 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:22.454727    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b639316-fe8e-40e3-bab9-19c4e66f9796" path="/var/lib/kubelet/pods/6b639316-fe8e-40e3-bab9-19c4e66f9796/volumes"
Oct 31 02:05:22.499082 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:22.499048    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:23.196754 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:23.196706    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:27.399966 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:27.399934    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:29.013881 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:29.013838    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:29.180652 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:05:29.180617    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:05:29.193310 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:29.193285    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:29.368991 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:29.368958    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:31.385378 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:31.385345    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:32.560509 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:32.560479    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:37.514930 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:37.514897    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:39.383222 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:39.383187    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:39.637881 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:39.637805    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:42.121086 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:42.121024    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:42.301927 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:42.301895    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:42.660075 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:42.660042    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:42.842699 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:42.842669    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:43.055313 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:43.055287    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:43.252298 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:43.252269    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:46.984666 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:46.984637    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:48.480163 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:48.480133    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:48.692034 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:48.692003    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:48.862784 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:48.862756    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:49.399981 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:49.399954    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:53.609718 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:53.609687    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:54.662780 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:54.662745    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:54.864982 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:54.864943    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:55.181921 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:55.181894    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:55.889109 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:55.889074    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:57.652922 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:57.652889    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:57.969718 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:57.969612    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:58.141878 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:58.141843    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:58.981113 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:05:58.981076    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:59.182123 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:05:59.182089    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:06:02.456418 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:02.456384    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:02.651269 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:02.651235    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:04.106593 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:04.106564    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:08.060238 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:08.060188    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:08.233280 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:08.233247    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:09.188254 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:09.188221    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:11.951631 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:11.951539    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"]
Oct 31 02:06:12.110348 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:12.110317    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:12.357691 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:12.357652    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:12.639560 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:12.639474    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:14.178027 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:14.177997    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:16.952554 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:16.952492    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8" podUID="3199cbdc-b6ba-4212-af9d-fd3d42a6f944" containerName="payload-processing" containerID="cri-o://397c67f9458712e368477562e5d565a31a99dee83a32d9dfab32fa126e59036a" gracePeriod=25
Oct 31 02:06:17.199841 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:17.199807    4476 generic.go:356] "Generic (PLEG): container finished" podID="3199cbdc-b6ba-4212-af9d-fd3d42a6f944" containerID="397c67f9458712e368477562e5d565a31a99dee83a32d9dfab32fa126e59036a" exitCode=0
Oct 31 02:06:17.199966 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:17.199875    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8" event={"ID":"3199cbdc-b6ba-4212-af9d-fd3d42a6f944","Type":"ContainerDied","Data":"397c67f9458712e368477562e5d565a31a99dee83a32d9dfab32fa126e59036a"}
Oct 31 02:06:17.199966 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:17.199953    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8" event={"ID":"3199cbdc-b6ba-4212-af9d-fd3d42a6f944","Type":"ContainerDied","Data":"b5089e67465c96fa96406009ad2ffd162fdfca07e36dbb7ab78b932baf4eec1e"}
Oct 31 02:06:17.200039 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:17.199972    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5089e67465c96fa96406009ad2ffd162fdfca07e36dbb7ab78b932baf4eec1e"
Oct 31 02:06:17.212328 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:17.212271    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"
Oct 31 02:06:17.339739 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:17.339705    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3199cbdc-b6ba-4212-af9d-fd3d42a6f944-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3199cbdc-b6ba-4212-af9d-fd3d42a6f944-plugins-config-volume\") pod \"3199cbdc-b6ba-4212-af9d-fd3d42a6f944\" (UID: \"3199cbdc-b6ba-4212-af9d-fd3d42a6f944\") "
Oct 31 02:06:17.339937 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:17.339808    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3199cbdc-b6ba-4212-af9d-fd3d42a6f944-kube-api-access-cnd9g\" (UniqueName: \"kubernetes.io/projected/3199cbdc-b6ba-4212-af9d-fd3d42a6f944-kube-api-access-cnd9g\") pod \"3199cbdc-b6ba-4212-af9d-fd3d42a6f944\" (UID: \"3199cbdc-b6ba-4212-af9d-fd3d42a6f944\") "
Oct 31 02:06:17.340092 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:17.340067    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3199cbdc-b6ba-4212-af9d-fd3d42a6f944-plugins-config-volume" pod "3199cbdc-b6ba-4212-af9d-fd3d42a6f944" (UID: "3199cbdc-b6ba-4212-af9d-fd3d42a6f944"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:06:17.340310 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:17.340293    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3199cbdc-b6ba-4212-af9d-fd3d42a6f944-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:17.342038 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:17.342011    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3199cbdc-b6ba-4212-af9d-fd3d42a6f944-kube-api-access-cnd9g" pod "3199cbdc-b6ba-4212-af9d-fd3d42a6f944" (UID: "3199cbdc-b6ba-4212-af9d-fd3d42a6f944"). InnerVolumeSpecName "kube-api-access-cnd9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:17.441704 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:17.441672    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnd9g\" (UniqueName: \"kubernetes.io/projected/3199cbdc-b6ba-4212-af9d-fd3d42a6f944-kube-api-access-cnd9g\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:18.206541 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:18.206509    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"
Oct 31 02:06:18.369555 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:18.369510    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"]
Oct 31 02:06:18.419230 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:18.419197    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fbe06f-97c787bd7-5tbv8"]
Oct 31 02:06:18.455548 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:18.455524    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3199cbdc-b6ba-4212-af9d-fd3d42a6f944" path="/var/lib/kubelet/pods/3199cbdc-b6ba-4212-af9d-fd3d42a6f944/volumes"
Oct 31 02:06:18.931200 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:18.931112    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:23.609853 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:23.609822    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:27.658815 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:27.658764    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:27.849705 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:27.849672    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:29.157440 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:29.157400    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:29.182713 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:06:29.182681    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:06:29.417042 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:29.416959    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:29.658390 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:29.658359    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:31.009764 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:31.009733    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 02:06:31.018417 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:31.018390    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 02:06:31.333835 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:31.333798    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:31.504818 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:31.504787    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:34.464521 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:34.464491    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:39.119271 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:39.119237    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:40.121012 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:40.120951    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:43.057014 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.056980    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"]
Oct 31 02:06:43.057501 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.057222    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj" podUID="4ea835fa-fab8-4d8b-9b94-7c2478ad41e9" containerName="maas-api" containerID="cri-o://68685eba9a9a39a151e0e2bb229ce010febc0727d451d1eed7aa85d2c483bd8b" gracePeriod=30
Oct 31 02:06:43.283169 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.283140    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:43.316286 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.316210    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:06:43.362266 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.362225    4476 generic.go:356] "Generic (PLEG): container finished" podID="4ea835fa-fab8-4d8b-9b94-7c2478ad41e9" containerID="68685eba9a9a39a151e0e2bb229ce010febc0727d451d1eed7aa85d2c483bd8b" exitCode=0
Oct 31 02:06:43.362434 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.362273    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"
Oct 31 02:06:43.362434 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.362309    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj" event={"ID":"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9","Type":"ContainerDied","Data":"68685eba9a9a39a151e0e2bb229ce010febc0727d451d1eed7aa85d2c483bd8b"}
Oct 31 02:06:43.362434 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.362366    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj" event={"ID":"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9","Type":"ContainerDied","Data":"836df75d356ae55e374b4b4b865e661729b0f0c130945eb76b52d39a3b359f8e"}
Oct 31 02:06:43.362434 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.362392    4476 scope.go:122] "RemoveContainer" containerID="68685eba9a9a39a151e0e2bb229ce010febc0727d451d1eed7aa85d2c483bd8b"
Oct 31 02:06:43.372210 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.372167    4476 scope.go:122] "RemoveContainer" containerID="68685eba9a9a39a151e0e2bb229ce010febc0727d451d1eed7aa85d2c483bd8b"
Oct 31 02:06:43.372479 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:06:43.372461    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68685eba9a9a39a151e0e2bb229ce010febc0727d451d1eed7aa85d2c483bd8b\": container with ID starting with 68685eba9a9a39a151e0e2bb229ce010febc0727d451d1eed7aa85d2c483bd8b not found: ID does not exist" containerID="68685eba9a9a39a151e0e2bb229ce010febc0727d451d1eed7aa85d2c483bd8b"
Oct 31 02:06:43.372534 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.372487    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68685eba9a9a39a151e0e2bb229ce010febc0727d451d1eed7aa85d2c483bd8b"} err="failed to get container status \"68685eba9a9a39a151e0e2bb229ce010febc0727d451d1eed7aa85d2c483bd8b\": rpc error: code = NotFound desc = could not find container \"68685eba9a9a39a151e0e2bb229ce010febc0727d451d1eed7aa85d2c483bd8b\": container with ID starting with 68685eba9a9a39a151e0e2bb229ce010febc0727d451d1eed7aa85d2c483bd8b not found: ID does not exist"
Oct 31 02:06:43.438567 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.438534    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-maas-api-tls\") pod \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") "
Oct 31 02:06:43.438717 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.438608    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-kube-api-access-5lntj\" (UniqueName: \"kubernetes.io/projected/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-kube-api-access-5lntj\") pod \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") "
Oct 31 02:06:43.438717 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.438650    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-openshift-service-ca\") pod \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") "
Oct 31 02:06:43.438878 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.438860    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-metrics-certs\") pod \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\" (UID: \"4ea835fa-fab8-4d8b-9b94-7c2478ad41e9\") "
Oct 31 02:06:43.439052 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.439028    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-openshift-service-ca" pod "4ea835fa-fab8-4d8b-9b94-7c2478ad41e9" (UID: "4ea835fa-fab8-4d8b-9b94-7c2478ad41e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:06:43.439453 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.439380    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:43.440894 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.440873    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-maas-api-tls" pod "4ea835fa-fab8-4d8b-9b94-7c2478ad41e9" (UID: "4ea835fa-fab8-4d8b-9b94-7c2478ad41e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:06:43.441091 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.441072    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-kube-api-access-5lntj" pod "4ea835fa-fab8-4d8b-9b94-7c2478ad41e9" (UID: "4ea835fa-fab8-4d8b-9b94-7c2478ad41e9"). InnerVolumeSpecName "kube-api-access-5lntj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:43.441294 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.441274    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-metrics-certs" pod "4ea835fa-fab8-4d8b-9b94-7c2478ad41e9" (UID: "4ea835fa-fab8-4d8b-9b94-7c2478ad41e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:06:43.465547 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.465518    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:43.540540 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.540510    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:43.540540 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.540539    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lntj\" (UniqueName: \"kubernetes.io/projected/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-kube-api-access-5lntj\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:43.540760 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.540554    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:43.783659 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.783620    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"]
Oct 31 02:06:43.786330 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:43.786305    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6c6990-668967998c-76qqj"]
Oct 31 02:06:44.454833 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:44.454798    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea835fa-fab8-4d8b-9b94-7c2478ad41e9" path="/var/lib/kubelet/pods/4ea835fa-fab8-4d8b-9b94-7c2478ad41e9/volumes"
Oct 31 02:06:44.749693 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:44.749617    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:45.123695 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:45.123661    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:46.346507 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:46.346444    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:46.525084 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:46.525048    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:47.031884 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:47.031856    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:49.617289 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:49.617254    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:51.199996 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:51.199966    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:51.580801 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:51.580772    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:51.769290 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:51.769251    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:51.956595 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:51.956517    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:51.969424 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:51.969398    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"]
Oct 31 02:06:52.155307 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:52.155267    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:52.339413 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:52.339376    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:56.824731 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:56.824701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:58.017946 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:58.017914    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:58.198506 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:58.198477    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:58.396508 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:58.396480    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:59.181366 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:06:59.181327    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:06:59.514771 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:59.514690    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:59.685070 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:06:59.685042    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:00.250440 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:00.250410    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:01.443660 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:01.443592    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:02.552575 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:02.552549    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:03.467490 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:03.467461    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:05.946114 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:05.946079    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:06.127139 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:06.127079    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:06.785265 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:06.785234    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:06.990129 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:06.990087    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp" podUID="fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" containerName="main" containerID="cri-o://ed25902217710808e743d699389cadc3594b3882090ff1a2dc9d359d089e838b" gracePeriod=45
Oct 31 02:07:07.268803 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.268777    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 02:07:07.307805 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.307753    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-kube-api-access-jl7db\" (UniqueName: \"kubernetes.io/projected/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-kube-api-access-jl7db\") pod \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") "
Oct 31 02:07:07.307994 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.307810    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-tls-certs\" (UniqueName: \"kubernetes.io/secret/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-tls-certs\") pod \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") "
Oct 31 02:07:07.307994 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.307877    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-model-cache\") pod \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") "
Oct 31 02:07:07.307994 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.307923    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-dshm\") pod \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") "
Oct 31 02:07:07.307994 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.307962    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-home\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-home\") pod \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") "
Oct 31 02:07:07.308278 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.308047    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-tmp-dir\") pod \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\" (UID: \"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194\") "
Oct 31 02:07:07.308980 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.308953    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-tmp-dir" pod "fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" (UID: "fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:07.309454 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.309427    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-model-cache" pod "fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" (UID: "fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:07.310882 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.309806    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-home" pod "fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" (UID: "fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:07.311583 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.311552    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-kube-api-access-jl7db" pod "fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" (UID: "fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194"). InnerVolumeSpecName "kube-api-access-jl7db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:07.311927 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.311902    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-tls-certs" pod "fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" (UID: "fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:07.313220 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.313186    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-dshm" pod "fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" (UID: "fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:07.410770 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.410693    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:07.410770 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.410735    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl7db\" (UniqueName: \"kubernetes.io/projected/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-kube-api-access-jl7db\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:07.410770 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.410751    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:07.410770 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.410769    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:07.411025 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.410784    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:07.411025 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.410798    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:07.526939 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.526907    4476 generic.go:356] "Generic (PLEG): container finished" podID="fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" containerID="ed25902217710808e743d699389cadc3594b3882090ff1a2dc9d359d089e838b" exitCode=0
Oct 31 02:07:07.527117 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.527001    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp" event={"ID":"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194","Type":"ContainerDied","Data":"ed25902217710808e743d699389cadc3594b3882090ff1a2dc9d359d089e838b"}
Oct 31 02:07:07.527117 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.527038    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp" event={"ID":"fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194","Type":"ContainerDied","Data":"ccb3ec40049a5652ef1f827762a84a43f2316b4b09db06179ebebaaf702da87a"}
Oct 31 02:07:07.527117 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.527059    4476 scope.go:122] "RemoveContainer" containerID="ed25902217710808e743d699389cadc3594b3882090ff1a2dc9d359d089e838b"
Oct 31 02:07:07.527353 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.527247    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"
Oct 31 02:07:07.538901 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.538878    4476 scope.go:122] "RemoveContainer" containerID="ed25902217710808e743d699389cadc3594b3882090ff1a2dc9d359d089e838b"
Oct 31 02:07:07.539281 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:07:07.539260    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed25902217710808e743d699389cadc3594b3882090ff1a2dc9d359d089e838b\": container with ID starting with ed25902217710808e743d699389cadc3594b3882090ff1a2dc9d359d089e838b not found: ID does not exist" containerID="ed25902217710808e743d699389cadc3594b3882090ff1a2dc9d359d089e838b"
Oct 31 02:07:07.539374 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.539285    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed25902217710808e743d699389cadc3594b3882090ff1a2dc9d359d089e838b"} err="failed to get container status \"ed25902217710808e743d699389cadc3594b3882090ff1a2dc9d359d089e838b\": rpc error: code = NotFound desc = could not find container \"ed25902217710808e743d699389cadc3594b3882090ff1a2dc9d359d089e838b\": container with ID starting with ed25902217710808e743d699389cadc3594b3882090ff1a2dc9d359d089e838b not found: ID does not exist"
Oct 31 02:07:07.694026 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.693950    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"]
Oct 31 02:07:07.698284 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:07.698264    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/simulator-w0-26fbc4-kserve-64b8598b6b-2h2kp"]
Oct 31 02:07:08.454424 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:08.454393    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" path="/var/lib/kubelet/pods/fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194/volumes"
Oct 31 02:07:11.073118 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.073088    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"]
Oct 31 02:07:11.073564 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.073297    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67" podUID="6af72007-50de-4d45-9c18-bdea122f32e6" containerName="maas-api" containerID="cri-o://93bebcc190025bfdf79e12d74d5820b7cb709cf42f391b6180e3cc63b8ca616d" gracePeriod=30
Oct 31 02:07:11.210837 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.210807    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:11.353822 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.353794    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:07:11.437447 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.437412    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:11.458647 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.458618    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af72007-50de-4d45-9c18-bdea122f32e6-kube-api-access-fkkbc\" (UniqueName: \"kubernetes.io/projected/6af72007-50de-4d45-9c18-bdea122f32e6-kube-api-access-fkkbc\") pod \"6af72007-50de-4d45-9c18-bdea122f32e6\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") "
Oct 31 02:07:11.458839 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.458711    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af72007-50de-4d45-9c18-bdea122f32e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af72007-50de-4d45-9c18-bdea122f32e6-openshift-service-ca\") pod \"6af72007-50de-4d45-9c18-bdea122f32e6\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") "
Oct 31 02:07:11.458839 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.458756    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-maas-api-tls\") pod \"6af72007-50de-4d45-9c18-bdea122f32e6\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") "
Oct 31 02:07:11.458839 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.458817    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-metrics-certs\") pod \"6af72007-50de-4d45-9c18-bdea122f32e6\" (UID: \"6af72007-50de-4d45-9c18-bdea122f32e6\") "
Oct 31 02:07:11.461976 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.461943    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af72007-50de-4d45-9c18-bdea122f32e6-openshift-service-ca" pod "6af72007-50de-4d45-9c18-bdea122f32e6" (UID: "6af72007-50de-4d45-9c18-bdea122f32e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:11.462123 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.462086    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-metrics-certs" pod "6af72007-50de-4d45-9c18-bdea122f32e6" (UID: "6af72007-50de-4d45-9c18-bdea122f32e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:11.462250 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.462229    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-maas-api-tls" pod "6af72007-50de-4d45-9c18-bdea122f32e6" (UID: "6af72007-50de-4d45-9c18-bdea122f32e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:11.462462 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.462424    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af72007-50de-4d45-9c18-bdea122f32e6-kube-api-access-fkkbc" pod "6af72007-50de-4d45-9c18-bdea122f32e6" (UID: "6af72007-50de-4d45-9c18-bdea122f32e6"). InnerVolumeSpecName "kube-api-access-fkkbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:11.551377 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.551347    4476 generic.go:356] "Generic (PLEG): container finished" podID="6af72007-50de-4d45-9c18-bdea122f32e6" containerID="93bebcc190025bfdf79e12d74d5820b7cb709cf42f391b6180e3cc63b8ca616d" exitCode=0
Oct 31 02:07:11.551564 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.551399    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"
Oct 31 02:07:11.551564 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.551444    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67" event={"ID":"6af72007-50de-4d45-9c18-bdea122f32e6","Type":"ContainerDied","Data":"93bebcc190025bfdf79e12d74d5820b7cb709cf42f391b6180e3cc63b8ca616d"}
Oct 31 02:07:11.551564 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.551511    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67" event={"ID":"6af72007-50de-4d45-9c18-bdea122f32e6","Type":"ContainerDied","Data":"9a212dcca2441089273fcbe7f7c845260b032400fbb3632ce5f560e6a31a89f4"}
Oct 31 02:07:11.551564 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.551541    4476 scope.go:122] "RemoveContainer" containerID="93bebcc190025bfdf79e12d74d5820b7cb709cf42f391b6180e3cc63b8ca616d"
Oct 31 02:07:11.560213 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.560190    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:11.560322 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.560214    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkkbc\" (UniqueName: \"kubernetes.io/projected/6af72007-50de-4d45-9c18-bdea122f32e6-kube-api-access-fkkbc\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:11.560322 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.560226    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6af72007-50de-4d45-9c18-bdea122f32e6-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:11.560322 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.560235    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6af72007-50de-4d45-9c18-bdea122f32e6-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:11.561630 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.561608    4476 scope.go:122] "RemoveContainer" containerID="93bebcc190025bfdf79e12d74d5820b7cb709cf42f391b6180e3cc63b8ca616d"
Oct 31 02:07:11.561892 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:07:11.561874    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bebcc190025bfdf79e12d74d5820b7cb709cf42f391b6180e3cc63b8ca616d\": container with ID starting with 93bebcc190025bfdf79e12d74d5820b7cb709cf42f391b6180e3cc63b8ca616d not found: ID does not exist" containerID="93bebcc190025bfdf79e12d74d5820b7cb709cf42f391b6180e3cc63b8ca616d"
Oct 31 02:07:11.561957 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.561900    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bebcc190025bfdf79e12d74d5820b7cb709cf42f391b6180e3cc63b8ca616d"} err="failed to get container status \"93bebcc190025bfdf79e12d74d5820b7cb709cf42f391b6180e3cc63b8ca616d\": rpc error: code = NotFound desc = could not find container \"93bebcc190025bfdf79e12d74d5820b7cb709cf42f391b6180e3cc63b8ca616d\": container with ID starting with 93bebcc190025bfdf79e12d74d5820b7cb709cf42f391b6180e3cc63b8ca616d not found: ID does not exist"
Oct 31 02:07:11.620271 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.620169    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:11.714242 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.714168    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"]
Oct 31 02:07:11.717726 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:11.717700    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce8dbb-7bb84ccc4d-58t67"]
Oct 31 02:07:12.454718 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:12.454672    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af72007-50de-4d45-9c18-bdea122f32e6" path="/var/lib/kubelet/pods/6af72007-50de-4d45-9c18-bdea122f32e6/volumes"
Oct 31 02:07:12.903110 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:12.903081    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:13.178003 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:13.177924    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:13.365152 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:13.365123    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:13.877824 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:13.877789    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:18.360704 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:18.360672    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:19.613438 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:19.613385    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:19.795790 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:19.795753    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:20.002958 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:20.002854    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:23.270981 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:23.270950    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:23.451813 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:23.451782    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:23.627739 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:23.627701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:28.063521 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:28.063488    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:28.255539 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:28.255504    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:29.197460 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:07:29.197419    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:07:29.343975 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:29.343945    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:29.532041 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:29.532004    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:29.714344 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:29.714311    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:29.896429 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:29.896352    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:30.085337 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:30.085305    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:30.486931 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:30.486891    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:31.461708 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:31.461676    4476 scope.go:122] "RemoveContainer" containerID="397c67f9458712e368477562e5d565a31a99dee83a32d9dfab32fa126e59036a"
Oct 31 02:07:32.222803 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:32.222771    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"]
Oct 31 02:07:32.330695 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:32.330667    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"]
Oct 31 02:07:32.661935 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:32.661904    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:32.836459 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:32.836425    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:34.142241 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.142203    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"]
Oct 31 02:07:34.142725 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.142511    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerName="maas-api" containerID="cri-o://e88621ec2a69b32dca465dc95436dc9c08d9c70fc6d06366b61bac272033f57b" gracePeriod=30
Oct 31 02:07:34.285354 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.285313    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:34.403320 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.403249    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:07:34.515761 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.515734    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26dc7e6-eb1f-4733-930f-836d9703d440-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26dc7e6-eb1f-4733-930f-836d9703d440-openshift-service-ca\") pod \"b26dc7e6-eb1f-4733-930f-836d9703d440\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") "
Oct 31 02:07:34.515917 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.515788    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-maas-api-tls\") pod \"b26dc7e6-eb1f-4733-930f-836d9703d440\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") "
Oct 31 02:07:34.515917 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.515818    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26dc7e6-eb1f-4733-930f-836d9703d440-kube-api-access-jbmgl\" (UniqueName: \"kubernetes.io/projected/b26dc7e6-eb1f-4733-930f-836d9703d440-kube-api-access-jbmgl\") pod \"b26dc7e6-eb1f-4733-930f-836d9703d440\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") "
Oct 31 02:07:34.516009 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.515925    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-metrics-certs\") pod \"b26dc7e6-eb1f-4733-930f-836d9703d440\" (UID: \"b26dc7e6-eb1f-4733-930f-836d9703d440\") "
Oct 31 02:07:34.516079 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.516055    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26dc7e6-eb1f-4733-930f-836d9703d440-openshift-service-ca" pod "b26dc7e6-eb1f-4733-930f-836d9703d440" (UID: "b26dc7e6-eb1f-4733-930f-836d9703d440"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:34.516406 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.516380    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26dc7e6-eb1f-4733-930f-836d9703d440-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:34.518158 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.518132    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26dc7e6-eb1f-4733-930f-836d9703d440-kube-api-access-jbmgl" pod "b26dc7e6-eb1f-4733-930f-836d9703d440" (UID: "b26dc7e6-eb1f-4733-930f-836d9703d440"). InnerVolumeSpecName "kube-api-access-jbmgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:34.518727 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.518702    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-metrics-certs" pod "b26dc7e6-eb1f-4733-930f-836d9703d440" (UID: "b26dc7e6-eb1f-4733-930f-836d9703d440"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:34.518816 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.518726    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-maas-api-tls" pod "b26dc7e6-eb1f-4733-930f-836d9703d440" (UID: "b26dc7e6-eb1f-4733-930f-836d9703d440"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:34.618005 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.617976    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:34.618005 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.618003    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b26dc7e6-eb1f-4733-930f-836d9703d440-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:34.618005 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.618012    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbmgl\" (UniqueName: \"kubernetes.io/projected/b26dc7e6-eb1f-4733-930f-836d9703d440-kube-api-access-jbmgl\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:34.690223 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.690142    4476 generic.go:356] "Generic (PLEG): container finished" podID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerID="e88621ec2a69b32dca465dc95436dc9c08d9c70fc6d06366b61bac272033f57b" exitCode=0
Oct 31 02:07:34.690223 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.690211    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"
Oct 31 02:07:34.690391 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.690230    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" event={"ID":"b26dc7e6-eb1f-4733-930f-836d9703d440","Type":"ContainerDied","Data":"e88621ec2a69b32dca465dc95436dc9c08d9c70fc6d06366b61bac272033f57b"}
Oct 31 02:07:34.690391 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.690288    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g" event={"ID":"b26dc7e6-eb1f-4733-930f-836d9703d440","Type":"ContainerDied","Data":"f2c3582287d648d2c23ab6056fdee708510564026aa5da22cd0a0e9c8381a510"}
Oct 31 02:07:34.690391 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.690310    4476 scope.go:122] "RemoveContainer" containerID="e88621ec2a69b32dca465dc95436dc9c08d9c70fc6d06366b61bac272033f57b"
Oct 31 02:07:34.702453 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.702430    4476 scope.go:122] "RemoveContainer" containerID="f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d"
Oct 31 02:07:34.711947 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.711931    4476 scope.go:122] "RemoveContainer" containerID="e88621ec2a69b32dca465dc95436dc9c08d9c70fc6d06366b61bac272033f57b"
Oct 31 02:07:34.712244 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:07:34.712222    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88621ec2a69b32dca465dc95436dc9c08d9c70fc6d06366b61bac272033f57b\": container with ID starting with e88621ec2a69b32dca465dc95436dc9c08d9c70fc6d06366b61bac272033f57b not found: ID does not exist" containerID="e88621ec2a69b32dca465dc95436dc9c08d9c70fc6d06366b61bac272033f57b"
Oct 31 02:07:34.712350 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.712253    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88621ec2a69b32dca465dc95436dc9c08d9c70fc6d06366b61bac272033f57b"} err="failed to get container status \"e88621ec2a69b32dca465dc95436dc9c08d9c70fc6d06366b61bac272033f57b\": rpc error: code = NotFound desc = could not find container \"e88621ec2a69b32dca465dc95436dc9c08d9c70fc6d06366b61bac272033f57b\": container with ID starting with e88621ec2a69b32dca465dc95436dc9c08d9c70fc6d06366b61bac272033f57b not found: ID does not exist"
Oct 31 02:07:34.712350 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.712270    4476 scope.go:122] "RemoveContainer" containerID="f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d"
Oct 31 02:07:34.712508 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:07:34.712492    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d\": container with ID starting with f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d not found: ID does not exist" containerID="f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d"
Oct 31 02:07:34.712550 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.712523    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d"} err="failed to get container status \"f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d\": rpc error: code = NotFound desc = could not find container \"f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d\": container with ID starting with f520ec2a146c8bb4d31c01987ae36890b3782d6117bfe381379c151687e43d6d not found: ID does not exist"
Oct 31 02:07:34.853273 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.853240    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"]
Oct 31 02:07:34.858621 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:34.858595    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-caff19-5878ccd45-mbw6g"]
Oct 31 02:07:35.987446 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:35.987415    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:36.192931 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:36.192903    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:36.388387 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:36.388357    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:36.454426 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:36.454388    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440" path="/var/lib/kubelet/pods/b26dc7e6-eb1f-4733-930f-836d9703d440/volumes"
Oct 31 02:07:36.581367 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:36.581340    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:36.865659 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:36.865627    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:37.223304 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.223208    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw" podUID="dcdec68c-a09e-45ac-8ed8-1aa397ccf660" containerName="payload-processing" containerID="cri-o://83e87720b3c62cf30962bc65e727b10e1d2067b7c9f96939847ad0155bffafed" gracePeriod=25
Oct 31 02:07:37.331099 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.331062    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr" podUID="3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14" containerName="payload-pre-processing" containerID="cri-o://94307c68bce61a83c9a45847fe7ed5dd02deb2ea5329f27870b41355d9205c1f" gracePeriod=25
Oct 31 02:07:37.488754 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.488693    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"
Oct 31 02:07:37.548554 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.548521    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcdec68c-a09e-45ac-8ed8-1aa397ccf660-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcdec68c-a09e-45ac-8ed8-1aa397ccf660-plugins-config-volume\") pod \"dcdec68c-a09e-45ac-8ed8-1aa397ccf660\" (UID: \"dcdec68c-a09e-45ac-8ed8-1aa397ccf660\") "
Oct 31 02:07:37.548794 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.548768    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcdec68c-a09e-45ac-8ed8-1aa397ccf660-kube-api-access-k9t9c\" (UniqueName: \"kubernetes.io/projected/dcdec68c-a09e-45ac-8ed8-1aa397ccf660-kube-api-access-k9t9c\") pod \"dcdec68c-a09e-45ac-8ed8-1aa397ccf660\" (UID: \"dcdec68c-a09e-45ac-8ed8-1aa397ccf660\") "
Oct 31 02:07:37.549321 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.549290    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcdec68c-a09e-45ac-8ed8-1aa397ccf660-plugins-config-volume" pod "dcdec68c-a09e-45ac-8ed8-1aa397ccf660" (UID: "dcdec68c-a09e-45ac-8ed8-1aa397ccf660"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:37.549454 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.549379    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcdec68c-a09e-45ac-8ed8-1aa397ccf660-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:37.551653 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.551625    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdec68c-a09e-45ac-8ed8-1aa397ccf660-kube-api-access-k9t9c" pod "dcdec68c-a09e-45ac-8ed8-1aa397ccf660" (UID: "dcdec68c-a09e-45ac-8ed8-1aa397ccf660"). InnerVolumeSpecName "kube-api-access-k9t9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:37.586983 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.586963    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"
Oct 31 02:07:37.650702 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.650673    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14-kube-api-access-4c2dm\" (UniqueName: \"kubernetes.io/projected/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14-kube-api-access-4c2dm\") pod \"3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14\" (UID: \"3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14\") "
Oct 31 02:07:37.650870 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.650808    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14-plugins-config-volume\") pod \"3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14\" (UID: \"3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14\") "
Oct 31 02:07:37.651235 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.651212    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14-plugins-config-volume" pod "3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14" (UID: "3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:37.651322 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.651219    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9t9c\" (UniqueName: \"kubernetes.io/projected/dcdec68c-a09e-45ac-8ed8-1aa397ccf660-kube-api-access-k9t9c\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:37.652864 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.652841    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14-kube-api-access-4c2dm" pod "3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14" (UID: "3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14"). InnerVolumeSpecName "kube-api-access-4c2dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:37.709598 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.709568    4476 generic.go:356] "Generic (PLEG): container finished" podID="dcdec68c-a09e-45ac-8ed8-1aa397ccf660" containerID="83e87720b3c62cf30962bc65e727b10e1d2067b7c9f96939847ad0155bffafed" exitCode=0
Oct 31 02:07:37.709712 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.709610    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"
Oct 31 02:07:37.709712 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.709647    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw" event={"ID":"dcdec68c-a09e-45ac-8ed8-1aa397ccf660","Type":"ContainerDied","Data":"83e87720b3c62cf30962bc65e727b10e1d2067b7c9f96939847ad0155bffafed"}
Oct 31 02:07:37.709712 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.709697    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw" event={"ID":"dcdec68c-a09e-45ac-8ed8-1aa397ccf660","Type":"ContainerDied","Data":"807b1d8b2a0558c92e731b9ab0881fcf61a9992cf4d513a21f0fe1512ade69ea"}
Oct 31 02:07:37.709864 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.709721    4476 scope.go:122] "RemoveContainer" containerID="83e87720b3c62cf30962bc65e727b10e1d2067b7c9f96939847ad0155bffafed"
Oct 31 02:07:37.711278 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.711253    4476 generic.go:356] "Generic (PLEG): container finished" podID="3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14" containerID="94307c68bce61a83c9a45847fe7ed5dd02deb2ea5329f27870b41355d9205c1f" exitCode=0
Oct 31 02:07:37.711615 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.711319    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"
Oct 31 02:07:37.711615 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.711337    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr" event={"ID":"3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14","Type":"ContainerDied","Data":"94307c68bce61a83c9a45847fe7ed5dd02deb2ea5329f27870b41355d9205c1f"}
Oct 31 02:07:37.711615 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.711372    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr" event={"ID":"3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14","Type":"ContainerDied","Data":"58db966fe6a1af6d2d46bb4ff1c0935ae8d17d2e11b15d409116e93c933395d2"}
Oct 31 02:07:37.722018 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.721999    4476 scope.go:122] "RemoveContainer" containerID="83e87720b3c62cf30962bc65e727b10e1d2067b7c9f96939847ad0155bffafed"
Oct 31 02:07:37.722387 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:07:37.722361    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e87720b3c62cf30962bc65e727b10e1d2067b7c9f96939847ad0155bffafed\": container with ID starting with 83e87720b3c62cf30962bc65e727b10e1d2067b7c9f96939847ad0155bffafed not found: ID does not exist" containerID="83e87720b3c62cf30962bc65e727b10e1d2067b7c9f96939847ad0155bffafed"
Oct 31 02:07:37.722509 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.722396    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e87720b3c62cf30962bc65e727b10e1d2067b7c9f96939847ad0155bffafed"} err="failed to get container status \"83e87720b3c62cf30962bc65e727b10e1d2067b7c9f96939847ad0155bffafed\": rpc error: code = NotFound desc = could not find container \"83e87720b3c62cf30962bc65e727b10e1d2067b7c9f96939847ad0155bffafed\": container with ID starting with 83e87720b3c62cf30962bc65e727b10e1d2067b7c9f96939847ad0155bffafed not found: ID does not exist"
Oct 31 02:07:37.722509 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.722422    4476 scope.go:122] "RemoveContainer" containerID="94307c68bce61a83c9a45847fe7ed5dd02deb2ea5329f27870b41355d9205c1f"
Oct 31 02:07:37.736325 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.733225    4476 scope.go:122] "RemoveContainer" containerID="94307c68bce61a83c9a45847fe7ed5dd02deb2ea5329f27870b41355d9205c1f"
Oct 31 02:07:37.736325 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:07:37.733563    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94307c68bce61a83c9a45847fe7ed5dd02deb2ea5329f27870b41355d9205c1f\": container with ID starting with 94307c68bce61a83c9a45847fe7ed5dd02deb2ea5329f27870b41355d9205c1f not found: ID does not exist" containerID="94307c68bce61a83c9a45847fe7ed5dd02deb2ea5329f27870b41355d9205c1f"
Oct 31 02:07:37.736325 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.733595    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94307c68bce61a83c9a45847fe7ed5dd02deb2ea5329f27870b41355d9205c1f"} err="failed to get container status \"94307c68bce61a83c9a45847fe7ed5dd02deb2ea5329f27870b41355d9205c1f\": rpc error: code = NotFound desc = could not find container \"94307c68bce61a83c9a45847fe7ed5dd02deb2ea5329f27870b41355d9205c1f\": container with ID starting with 94307c68bce61a83c9a45847fe7ed5dd02deb2ea5329f27870b41355d9205c1f not found: ID does not exist"
Oct 31 02:07:37.752003 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.751939    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c2dm\" (UniqueName: \"kubernetes.io/projected/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14-kube-api-access-4c2dm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:37.752003 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.751967    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:37.929860 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.929825    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"]
Oct 31 02:07:37.932392 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:37.932371    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-caff19-58756c467c-mzgnw"]
Oct 31 02:07:38.038419 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:38.038389    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"]
Oct 31 02:07:38.041246 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:38.041226    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-caff19-7864c8b6cf-ndwwr"]
Oct 31 02:07:38.462464 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:38.462390    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14" path="/var/lib/kubelet/pods/3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14/volumes"
Oct 31 02:07:38.462835 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:38.462758    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcdec68c-a09e-45ac-8ed8-1aa397ccf660" path="/var/lib/kubelet/pods/dcdec68c-a09e-45ac-8ed8-1aa397ccf660/volumes"
Oct 31 02:07:41.433992 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:41.433908    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:43.490244 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:43.490206    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:46.350492 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:46.350455    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:49.644591 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:49.644559    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:49.818744 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:49.818711    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:51.026521 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:51.026488    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:55.793478 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:07:55.793441    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:59.181439 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:07:59.181404    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:08:00.664432 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:00.664347    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:05.364134 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:05.364103    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:10.307845 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:10.307806    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:15.090774 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:15.090738    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:17.456745 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:17.456710    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:19.166461 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:19.166427    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:20.684451 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:20.684414    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:20.894913 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:20.894877    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:21.133037 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:21.132997    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:21.659734 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:21.659692    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:26.096791 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:26.096755    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:27.513159 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:27.513126    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:27.698059 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:27.698026    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:27.922620 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:27.922578    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:28.558494 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:28.558463    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:29.181736 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:08:29.181696    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:08:31.296809 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:31.296777    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:36.195494 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:36.195462    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:40.902832 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:40.902802    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:42.756528 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:42.756494    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"]
Oct 31 02:08:43.063246 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:43.063206    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:43.285954 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:43.285922    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:43.828139 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:43.828097    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:44.141161 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:44.140423    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:45.647271 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:45.647232    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:46.401944 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:46.401911    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:46.585254 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:46.585218    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:46.997162 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:46.997128    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:47.217835 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:47.217806    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:47.402350 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:47.402312    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:47.757802 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:47.757722    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t" podUID="ae3e94c6-c7e3-4e24-a832-4e632806f191" containerName="payload-pre-processing" containerID="cri-o://5be17606205e439b003c9ad57e27e9fa13724ae73f3fa75f94cea6a476385c42" gracePeriod=25
Oct 31 02:08:47.987680 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:47.987649    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:51.079796 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:51.079761    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:51.274457 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:51.274428    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:52.248571 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:52.248536    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:53.077744 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:53.077705    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t" podUID="ae3e94c6-c7e3-4e24-a832-4e632806f191" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:08:53.858224 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:53.858167    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:54.044303 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:54.044269    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:54.245602 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:54.245515    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:54.468556 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:54.468524    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:54.674710 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:54.674679    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:56.812550 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:56.812513    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:58.398808 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:58.398778    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:58.662042 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:58.661960    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:58.848138 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:58.848102    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:59.182507 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:08:59.182468    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:08:59.330450 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:08:59.330391    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:03.076752 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:03.076709    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t" podUID="ae3e94c6-c7e3-4e24-a832-4e632806f191" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:09:03.762078 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:03.762040    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:05.293408 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:05.293371    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:05.473851 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:05.473820    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:05.666411 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:05.666326    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:06.641331 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:06.641291    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:06.827061 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:06.827026    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:08.043487 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.043458    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"
Oct 31 02:09:08.157408 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.157378    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3e94c6-c7e3-4e24-a832-4e632806f191-kube-api-access-f52bd\" (UniqueName: \"kubernetes.io/projected/ae3e94c6-c7e3-4e24-a832-4e632806f191-kube-api-access-f52bd\") pod \"ae3e94c6-c7e3-4e24-a832-4e632806f191\" (UID: \"ae3e94c6-c7e3-4e24-a832-4e632806f191\") "
Oct 31 02:09:08.157408 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.157414    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae3e94c6-c7e3-4e24-a832-4e632806f191-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3e94c6-c7e3-4e24-a832-4e632806f191-plugins-config-volume\") pod \"ae3e94c6-c7e3-4e24-a832-4e632806f191\" (UID: \"ae3e94c6-c7e3-4e24-a832-4e632806f191\") "
Oct 31 02:09:08.157850 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.157823    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3e94c6-c7e3-4e24-a832-4e632806f191-plugins-config-volume" pod "ae3e94c6-c7e3-4e24-a832-4e632806f191" (UID: "ae3e94c6-c7e3-4e24-a832-4e632806f191"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:08.159878 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.159846    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3e94c6-c7e3-4e24-a832-4e632806f191-kube-api-access-f52bd" pod "ae3e94c6-c7e3-4e24-a832-4e632806f191" (UID: "ae3e94c6-c7e3-4e24-a832-4e632806f191"). InnerVolumeSpecName "kube-api-access-f52bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:08.257425 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.257393    4476 generic.go:356] "Generic (PLEG): container finished" podID="ae3e94c6-c7e3-4e24-a832-4e632806f191" containerID="5be17606205e439b003c9ad57e27e9fa13724ae73f3fa75f94cea6a476385c42" exitCode=0
Oct 31 02:09:08.257602 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.257444    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"
Oct 31 02:09:08.257602 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.257473    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t" event={"ID":"ae3e94c6-c7e3-4e24-a832-4e632806f191","Type":"ContainerDied","Data":"5be17606205e439b003c9ad57e27e9fa13724ae73f3fa75f94cea6a476385c42"}
Oct 31 02:09:08.257602 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.257539    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t" event={"ID":"ae3e94c6-c7e3-4e24-a832-4e632806f191","Type":"ContainerDied","Data":"aefc44844575e98516061b262d449056b4e18af81242b0a1357f487be891ac3e"}
Oct 31 02:09:08.257602 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.257568    4476 scope.go:122] "RemoveContainer" containerID="5be17606205e439b003c9ad57e27e9fa13724ae73f3fa75f94cea6a476385c42"
Oct 31 02:09:08.258651 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.258618    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f52bd\" (UniqueName: \"kubernetes.io/projected/ae3e94c6-c7e3-4e24-a832-4e632806f191-kube-api-access-f52bd\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:08.258651 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.258640    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3e94c6-c7e3-4e24-a832-4e632806f191-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:08.268414 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.268402    4476 scope.go:122] "RemoveContainer" containerID="5be17606205e439b003c9ad57e27e9fa13724ae73f3fa75f94cea6a476385c42"
Oct 31 02:09:08.268726 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:09:08.268690    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be17606205e439b003c9ad57e27e9fa13724ae73f3fa75f94cea6a476385c42\": container with ID starting with 5be17606205e439b003c9ad57e27e9fa13724ae73f3fa75f94cea6a476385c42 not found: ID does not exist" containerID="5be17606205e439b003c9ad57e27e9fa13724ae73f3fa75f94cea6a476385c42"
Oct 31 02:09:08.268812 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.268721    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be17606205e439b003c9ad57e27e9fa13724ae73f3fa75f94cea6a476385c42"} err="failed to get container status \"5be17606205e439b003c9ad57e27e9fa13724ae73f3fa75f94cea6a476385c42\": rpc error: code = NotFound desc = could not find container \"5be17606205e439b003c9ad57e27e9fa13724ae73f3fa75f94cea6a476385c42\": container with ID starting with 5be17606205e439b003c9ad57e27e9fa13724ae73f3fa75f94cea6a476385c42 not found: ID does not exist"
Oct 31 02:09:08.441404 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.441371    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"]
Oct 31 02:09:08.444286 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.444263    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:08.470864 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:08.470837    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b992fe-69564dfbd7-jj44t"]
Oct 31 02:09:10.015562 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:10.015526    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:10.390619 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:10.390588    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:10.460828 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:10.460797    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3e94c6-c7e3-4e24-a832-4e632806f191" path="/var/lib/kubelet/pods/ae3e94c6-c7e3-4e24-a832-4e632806f191/volumes"
Oct 31 02:09:12.453407 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:12.453378    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:13.388505 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:13.388469    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:15.715078 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:15.715049    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:18.513536 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:18.513497    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"]
Oct 31 02:09:18.616921 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:18.616890    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"]
Oct 31 02:09:18.912040 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:18.912005    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:19.087314 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:19.087282    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:19.573478 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:19.573441    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"]
Oct 31 02:09:19.573921 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:19.573653    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk" podUID="ef906818-75f2-40e3-a4b1-16f6f5e40a70" containerName="maas-api" containerID="cri-o://e26a242f08870ac6e7b747ad82767d94393e133ee0bb06bbf2498e88fb4ea809" gracePeriod=30
Oct 31 02:09:19.728719 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:19.728690    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:19.831141 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:19.831073    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 02:09:19.989066 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:19.988561    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef906818-75f2-40e3-a4b1-16f6f5e40a70-kube-api-access-2bsx6\" (UniqueName: \"kubernetes.io/projected/ef906818-75f2-40e3-a4b1-16f6f5e40a70-kube-api-access-2bsx6\") pod \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") "
Oct 31 02:09:19.989066 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:19.988684    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-metrics-certs\") pod \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") "
Oct 31 02:09:19.989066 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:19.988743    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-maas-api-tls\") pod \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") "
Oct 31 02:09:19.989066 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:19.988771    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef906818-75f2-40e3-a4b1-16f6f5e40a70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef906818-75f2-40e3-a4b1-16f6f5e40a70-openshift-service-ca\") pod \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\" (UID: \"ef906818-75f2-40e3-a4b1-16f6f5e40a70\") "
Oct 31 02:09:19.999601 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:19.999564    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef906818-75f2-40e3-a4b1-16f6f5e40a70-kube-api-access-2bsx6" pod "ef906818-75f2-40e3-a4b1-16f6f5e40a70" (UID: "ef906818-75f2-40e3-a4b1-16f6f5e40a70"). InnerVolumeSpecName "kube-api-access-2bsx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:20.000472 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.000440    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-maas-api-tls" pod "ef906818-75f2-40e3-a4b1-16f6f5e40a70" (UID: "ef906818-75f2-40e3-a4b1-16f6f5e40a70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:20.003318 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.003282    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef906818-75f2-40e3-a4b1-16f6f5e40a70-openshift-service-ca" pod "ef906818-75f2-40e3-a4b1-16f6f5e40a70" (UID: "ef906818-75f2-40e3-a4b1-16f6f5e40a70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:20.007576 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.007552    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-metrics-certs" pod "ef906818-75f2-40e3-a4b1-16f6f5e40a70" (UID: "ef906818-75f2-40e3-a4b1-16f6f5e40a70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:20.090170 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.090101    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bsx6\" (UniqueName: \"kubernetes.io/projected/ef906818-75f2-40e3-a4b1-16f6f5e40a70-kube-api-access-2bsx6\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:20.090170 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.090128    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:20.090170 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.090137    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef906818-75f2-40e3-a4b1-16f6f5e40a70-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:20.090170 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.090146    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef906818-75f2-40e3-a4b1-16f6f5e40a70-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:20.326824 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.326787    4476 generic.go:356] "Generic (PLEG): container finished" podID="ef906818-75f2-40e3-a4b1-16f6f5e40a70" containerID="e26a242f08870ac6e7b747ad82767d94393e133ee0bb06bbf2498e88fb4ea809" exitCode=0
Oct 31 02:09:20.327023 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.326838    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"
Oct 31 02:09:20.327023 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.326867    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk" event={"ID":"ef906818-75f2-40e3-a4b1-16f6f5e40a70","Type":"ContainerDied","Data":"e26a242f08870ac6e7b747ad82767d94393e133ee0bb06bbf2498e88fb4ea809"}
Oct 31 02:09:20.327023 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.326918    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk" event={"ID":"ef906818-75f2-40e3-a4b1-16f6f5e40a70","Type":"ContainerDied","Data":"cafcaa8425375538f8bb6b4718b86fea51fbc1a3bfe77180ec84bdee9482380c"}
Oct 31 02:09:20.327023 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.326938    4476 scope.go:122] "RemoveContainer" containerID="e26a242f08870ac6e7b747ad82767d94393e133ee0bb06bbf2498e88fb4ea809"
Oct 31 02:09:20.336802 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.336785    4476 scope.go:122] "RemoveContainer" containerID="e26a242f08870ac6e7b747ad82767d94393e133ee0bb06bbf2498e88fb4ea809"
Oct 31 02:09:20.337089 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:09:20.337072    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26a242f08870ac6e7b747ad82767d94393e133ee0bb06bbf2498e88fb4ea809\": container with ID starting with e26a242f08870ac6e7b747ad82767d94393e133ee0bb06bbf2498e88fb4ea809 not found: ID does not exist" containerID="e26a242f08870ac6e7b747ad82767d94393e133ee0bb06bbf2498e88fb4ea809"
Oct 31 02:09:20.337148 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.337101    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26a242f08870ac6e7b747ad82767d94393e133ee0bb06bbf2498e88fb4ea809"} err="failed to get container status \"e26a242f08870ac6e7b747ad82767d94393e133ee0bb06bbf2498e88fb4ea809\": rpc error: code = NotFound desc = could not find container \"e26a242f08870ac6e7b747ad82767d94393e133ee0bb06bbf2498e88fb4ea809\": container with ID starting with e26a242f08870ac6e7b747ad82767d94393e133ee0bb06bbf2498e88fb4ea809 not found: ID does not exist"
Oct 31 02:09:20.494643 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.494556    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"]
Oct 31 02:09:20.545115 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:20.545080    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-26fbc4-5c56d4d494-rxcnk"]
Oct 31 02:09:21.006800 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:21.006767    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:21.293554 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:21.293510    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:21.529971 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:21.529942    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:21.737958 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:21.737862    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:22.291511 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:22.291482    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:22.454808 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:22.454774    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef906818-75f2-40e3-a4b1-16f6f5e40a70" path="/var/lib/kubelet/pods/ef906818-75f2-40e3-a4b1-16f6f5e40a70/volumes"
Oct 31 02:09:23.514003 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:23.513965    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2" podUID="b14aee9c-3186-4e2a-9c84-9b50cb844194" containerName="payload-processing" containerID="cri-o://1be306d582c8a005b4d5b7b5b35bea6d795a4252fe9e349058bf69bd182c3931" gracePeriod=25
Oct 31 02:09:23.617838 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:23.617802    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5" podUID="0beaabcf-d87c-4769-9a19-84e9124568a3" containerName="payload-pre-processing" containerID="cri-o://a12a183319309ba46cbff46237de987a256480ec08cb386f9c3c25dad1b2e884" gracePeriod=25
Oct 31 02:09:26.570950 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:26.570917    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:26.885265 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:26.885168    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:27.064220 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:27.064168    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:27.266122 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:27.266023    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:28.537240 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:28.537164    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:28.725062 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:28.725030    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:28.919246 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:28.919113    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:29.181507 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:09:29.181421    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:09:30.659750 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:30.659708    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5" podUID="0beaabcf-d87c-4769-9a19-84e9124568a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:09:30.660270 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:30.659766    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2" podUID="b14aee9c-3186-4e2a-9c84-9b50cb844194" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:09:31.895945 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:31.895841    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:35.954150 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:35.954118    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"]
Oct 31 02:09:35.954619 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:35.954351    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc" podUID="810203c9-65fd-4958-a46e-7ecf24a22f41" containerName="maas-api" containerID="cri-o://8ccd014fade3ed542bff2d40e43fc915a8c252377232c75d1d9c7dd11d3d309c" gracePeriod=30
Oct 31 02:09:36.079925 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.079900    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:36.225902 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.225874    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 02:09:36.270054 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.270016    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/810203c9-65fd-4958-a46e-7ecf24a22f41-kube-api-access-qqs5m\" (UniqueName: \"kubernetes.io/projected/810203c9-65fd-4958-a46e-7ecf24a22f41-kube-api-access-qqs5m\") pod \"810203c9-65fd-4958-a46e-7ecf24a22f41\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") "
Oct 31 02:09:36.270238 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.270077    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/810203c9-65fd-4958-a46e-7ecf24a22f41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/810203c9-65fd-4958-a46e-7ecf24a22f41-openshift-service-ca\") pod \"810203c9-65fd-4958-a46e-7ecf24a22f41\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") "
Oct 31 02:09:36.270238 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.270115    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-maas-api-tls\") pod \"810203c9-65fd-4958-a46e-7ecf24a22f41\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") "
Oct 31 02:09:36.270340 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.270267    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-metrics-certs\") pod \"810203c9-65fd-4958-a46e-7ecf24a22f41\" (UID: \"810203c9-65fd-4958-a46e-7ecf24a22f41\") "
Oct 31 02:09:36.270507 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.270479    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/810203c9-65fd-4958-a46e-7ecf24a22f41-openshift-service-ca" pod "810203c9-65fd-4958-a46e-7ecf24a22f41" (UID: "810203c9-65fd-4958-a46e-7ecf24a22f41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:36.270754 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.270732    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/810203c9-65fd-4958-a46e-7ecf24a22f41-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:36.272401 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.272376    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-maas-api-tls" pod "810203c9-65fd-4958-a46e-7ecf24a22f41" (UID: "810203c9-65fd-4958-a46e-7ecf24a22f41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:36.272488 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.272401    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-metrics-certs" pod "810203c9-65fd-4958-a46e-7ecf24a22f41" (UID: "810203c9-65fd-4958-a46e-7ecf24a22f41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:36.272488 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.272414    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/810203c9-65fd-4958-a46e-7ecf24a22f41-kube-api-access-qqs5m" pod "810203c9-65fd-4958-a46e-7ecf24a22f41" (UID: "810203c9-65fd-4958-a46e-7ecf24a22f41"). InnerVolumeSpecName "kube-api-access-qqs5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:36.371805 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.371769    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqs5m\" (UniqueName: \"kubernetes.io/projected/810203c9-65fd-4958-a46e-7ecf24a22f41-kube-api-access-qqs5m\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:36.371805 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.371809    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:36.372014 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.371827    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810203c9-65fd-4958-a46e-7ecf24a22f41-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:36.420035 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.419998    4476 generic.go:356] "Generic (PLEG): container finished" podID="810203c9-65fd-4958-a46e-7ecf24a22f41" containerID="8ccd014fade3ed542bff2d40e43fc915a8c252377232c75d1d9c7dd11d3d309c" exitCode=0
Oct 31 02:09:36.420239 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.420044    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"
Oct 31 02:09:36.420239 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.420082    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc" event={"ID":"810203c9-65fd-4958-a46e-7ecf24a22f41","Type":"ContainerDied","Data":"8ccd014fade3ed542bff2d40e43fc915a8c252377232c75d1d9c7dd11d3d309c"}
Oct 31 02:09:36.420239 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.420138    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc" event={"ID":"810203c9-65fd-4958-a46e-7ecf24a22f41","Type":"ContainerDied","Data":"f3b064611f4ee174a4b533c1e02827fac7e1badbd73d102ef326cf9b778b51ca"}
Oct 31 02:09:36.420239 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.420168    4476 scope.go:122] "RemoveContainer" containerID="8ccd014fade3ed542bff2d40e43fc915a8c252377232c75d1d9c7dd11d3d309c"
Oct 31 02:09:36.430346 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.430326    4476 scope.go:122] "RemoveContainer" containerID="8ccd014fade3ed542bff2d40e43fc915a8c252377232c75d1d9c7dd11d3d309c"
Oct 31 02:09:36.430639 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:09:36.430620    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ccd014fade3ed542bff2d40e43fc915a8c252377232c75d1d9c7dd11d3d309c\": container with ID starting with 8ccd014fade3ed542bff2d40e43fc915a8c252377232c75d1d9c7dd11d3d309c not found: ID does not exist" containerID="8ccd014fade3ed542bff2d40e43fc915a8c252377232c75d1d9c7dd11d3d309c"
Oct 31 02:09:36.430683 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.430648    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ccd014fade3ed542bff2d40e43fc915a8c252377232c75d1d9c7dd11d3d309c"} err="failed to get container status \"8ccd014fade3ed542bff2d40e43fc915a8c252377232c75d1d9c7dd11d3d309c\": rpc error: code = NotFound desc = could not find container \"8ccd014fade3ed542bff2d40e43fc915a8c252377232c75d1d9c7dd11d3d309c\": container with ID starting with 8ccd014fade3ed542bff2d40e43fc915a8c252377232c75d1d9c7dd11d3d309c not found: ID does not exist"
Oct 31 02:09:36.582874 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.582842    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"]
Oct 31 02:09:36.587008 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:36.586976    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc9e63-5c59b9dcb6-7m4tc"]
Oct 31 02:09:37.567045 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:37.567010    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:37.780475 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:37.780442    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:37.963463 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:37.963373    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:38.454409 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:38.454373    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="810203c9-65fd-4958-a46e-7ecf24a22f41" path="/var/lib/kubelet/pods/810203c9-65fd-4958-a46e-7ecf24a22f41/volumes"
Oct 31 02:09:38.476459 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:38.476424    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:39.587145 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:39.587102    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"]
Oct 31 02:09:40.016896 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:40.016799    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:40.206486 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:40.206449    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:40.660477 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:40.660441    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5" podUID="0beaabcf-d87c-4769-9a19-84e9124568a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:09:40.660477 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:40.660463    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2" podUID="b14aee9c-3186-4e2a-9c84-9b50cb844194" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:09:42.997083 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:42.997047    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:43.793099 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:43.793064    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"
Oct 31 02:09:43.863676 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:43.863647    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14aee9c-3186-4e2a-9c84-9b50cb844194-kube-api-access-ll4jz\" (UniqueName: \"kubernetes.io/projected/b14aee9c-3186-4e2a-9c84-9b50cb844194-kube-api-access-ll4jz\") pod \"b14aee9c-3186-4e2a-9c84-9b50cb844194\" (UID: \"b14aee9c-3186-4e2a-9c84-9b50cb844194\") "
Oct 31 02:09:43.863802 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:43.863768    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b14aee9c-3186-4e2a-9c84-9b50cb844194-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b14aee9c-3186-4e2a-9c84-9b50cb844194-plugins-config-volume\") pod \"b14aee9c-3186-4e2a-9c84-9b50cb844194\" (UID: \"b14aee9c-3186-4e2a-9c84-9b50cb844194\") "
Oct 31 02:09:43.864253 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:43.864222    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b14aee9c-3186-4e2a-9c84-9b50cb844194-plugins-config-volume" pod "b14aee9c-3186-4e2a-9c84-9b50cb844194" (UID: "b14aee9c-3186-4e2a-9c84-9b50cb844194"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:43.866035 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:43.866008    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14aee9c-3186-4e2a-9c84-9b50cb844194-kube-api-access-ll4jz" pod "b14aee9c-3186-4e2a-9c84-9b50cb844194" (UID: "b14aee9c-3186-4e2a-9c84-9b50cb844194"). InnerVolumeSpecName "kube-api-access-ll4jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:43.878626 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:43.878604    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"
Oct 31 02:09:43.964601 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:43.964573    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0beaabcf-d87c-4769-9a19-84e9124568a3-kube-api-access-94n9j\" (UniqueName: \"kubernetes.io/projected/0beaabcf-d87c-4769-9a19-84e9124568a3-kube-api-access-94n9j\") pod \"0beaabcf-d87c-4769-9a19-84e9124568a3\" (UID: \"0beaabcf-d87c-4769-9a19-84e9124568a3\") "
Oct 31 02:09:43.964790 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:43.964737    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0beaabcf-d87c-4769-9a19-84e9124568a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0beaabcf-d87c-4769-9a19-84e9124568a3-plugins-config-volume\") pod \"0beaabcf-d87c-4769-9a19-84e9124568a3\" (UID: \"0beaabcf-d87c-4769-9a19-84e9124568a3\") "
Oct 31 02:09:43.964969 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:43.964957    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b14aee9c-3186-4e2a-9c84-9b50cb844194-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:43.965027 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:43.964970    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll4jz\" (UniqueName: \"kubernetes.io/projected/b14aee9c-3186-4e2a-9c84-9b50cb844194-kube-api-access-ll4jz\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:43.965092 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:43.965068    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0beaabcf-d87c-4769-9a19-84e9124568a3-plugins-config-volume" pod "0beaabcf-d87c-4769-9a19-84e9124568a3" (UID: "0beaabcf-d87c-4769-9a19-84e9124568a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:43.966883 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:43.966861    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0beaabcf-d87c-4769-9a19-84e9124568a3-kube-api-access-94n9j" pod "0beaabcf-d87c-4769-9a19-84e9124568a3" (UID: "0beaabcf-d87c-4769-9a19-84e9124568a3"). InnerVolumeSpecName "kube-api-access-94n9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:44.066103 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.066009    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94n9j\" (UniqueName: \"kubernetes.io/projected/0beaabcf-d87c-4769-9a19-84e9124568a3-kube-api-access-94n9j\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:44.066103 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.066041    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0beaabcf-d87c-4769-9a19-84e9124568a3-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:44.467358 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.467329    4476 generic.go:356] "Generic (PLEG): container finished" podID="b14aee9c-3186-4e2a-9c84-9b50cb844194" containerID="1be306d582c8a005b4d5b7b5b35bea6d795a4252fe9e349058bf69bd182c3931" exitCode=0
Oct 31 02:09:44.467489 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.467372    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"
Oct 31 02:09:44.467489 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.467405    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2" event={"ID":"b14aee9c-3186-4e2a-9c84-9b50cb844194","Type":"ContainerDied","Data":"1be306d582c8a005b4d5b7b5b35bea6d795a4252fe9e349058bf69bd182c3931"}
Oct 31 02:09:44.467489 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.467461    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2" event={"ID":"b14aee9c-3186-4e2a-9c84-9b50cb844194","Type":"ContainerDied","Data":"24f33506d4f1ed9449900f3fe72d4091d5e2479dcebd50ad2d34254ede343e4d"}
Oct 31 02:09:44.467644 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.467489    4476 scope.go:122] "RemoveContainer" containerID="1be306d582c8a005b4d5b7b5b35bea6d795a4252fe9e349058bf69bd182c3931"
Oct 31 02:09:44.469026 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.469004    4476 generic.go:356] "Generic (PLEG): container finished" podID="0beaabcf-d87c-4769-9a19-84e9124568a3" containerID="a12a183319309ba46cbff46237de987a256480ec08cb386f9c3c25dad1b2e884" exitCode=0
Oct 31 02:09:44.469137 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.469078    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5" event={"ID":"0beaabcf-d87c-4769-9a19-84e9124568a3","Type":"ContainerDied","Data":"a12a183319309ba46cbff46237de987a256480ec08cb386f9c3c25dad1b2e884"}
Oct 31 02:09:44.469137 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.469114    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5" event={"ID":"0beaabcf-d87c-4769-9a19-84e9124568a3","Type":"ContainerDied","Data":"93bb376fccc4224b9a3458b14381562b09fdb8d12d3c3a524fa5c3435e231dc3"}
Oct 31 02:09:44.469275 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.469213    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"
Oct 31 02:09:44.480710 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.480688    4476 scope.go:122] "RemoveContainer" containerID="1be306d582c8a005b4d5b7b5b35bea6d795a4252fe9e349058bf69bd182c3931"
Oct 31 02:09:44.480998 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:09:44.480972    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be306d582c8a005b4d5b7b5b35bea6d795a4252fe9e349058bf69bd182c3931\": container with ID starting with 1be306d582c8a005b4d5b7b5b35bea6d795a4252fe9e349058bf69bd182c3931 not found: ID does not exist" containerID="1be306d582c8a005b4d5b7b5b35bea6d795a4252fe9e349058bf69bd182c3931"
Oct 31 02:09:44.481056 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.481012    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be306d582c8a005b4d5b7b5b35bea6d795a4252fe9e349058bf69bd182c3931"} err="failed to get container status \"1be306d582c8a005b4d5b7b5b35bea6d795a4252fe9e349058bf69bd182c3931\": rpc error: code = NotFound desc = could not find container \"1be306d582c8a005b4d5b7b5b35bea6d795a4252fe9e349058bf69bd182c3931\": container with ID starting with 1be306d582c8a005b4d5b7b5b35bea6d795a4252fe9e349058bf69bd182c3931 not found: ID does not exist"
Oct 31 02:09:44.481056 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.481032    4476 scope.go:122] "RemoveContainer" containerID="a12a183319309ba46cbff46237de987a256480ec08cb386f9c3c25dad1b2e884"
Oct 31 02:09:44.490130 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.490112    4476 scope.go:122] "RemoveContainer" containerID="a12a183319309ba46cbff46237de987a256480ec08cb386f9c3c25dad1b2e884"
Oct 31 02:09:44.490439 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:09:44.490419    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12a183319309ba46cbff46237de987a256480ec08cb386f9c3c25dad1b2e884\": container with ID starting with a12a183319309ba46cbff46237de987a256480ec08cb386f9c3c25dad1b2e884 not found: ID does not exist" containerID="a12a183319309ba46cbff46237de987a256480ec08cb386f9c3c25dad1b2e884"
Oct 31 02:09:44.490489 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.490450    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12a183319309ba46cbff46237de987a256480ec08cb386f9c3c25dad1b2e884"} err="failed to get container status \"a12a183319309ba46cbff46237de987a256480ec08cb386f9c3c25dad1b2e884\": rpc error: code = NotFound desc = could not find container \"a12a183319309ba46cbff46237de987a256480ec08cb386f9c3c25dad1b2e884\": container with ID starting with a12a183319309ba46cbff46237de987a256480ec08cb386f9c3c25dad1b2e884 not found: ID does not exist"
Oct 31 02:09:44.587828 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.587781    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l" podUID="5efb0ead-026d-472a-b36d-8ad0cccd3487" containerName="payload-processing" containerID="cri-o://705593f2bb7d51b9ea6b91231dc51de412afd5fac74f7d82b86c82f74dbc8217" gracePeriod=25
Oct 31 02:09:44.636909 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.636878    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"]
Oct 31 02:09:44.640966 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.640936    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-083e17-58cf7b7b45-zxrm2"]
Oct 31 02:09:44.749446 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.749365    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"]
Oct 31 02:09:44.753985 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.753955    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-083e17-766fd46f8c-jstb5"]
Oct 31 02:09:44.904970 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:44.904940    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:45.100446 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:45.100415    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:45.475763 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:45.475679    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:46.454513 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:46.454413    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0beaabcf-d87c-4769-9a19-84e9124568a3" path="/var/lib/kubelet/pods/0beaabcf-d87c-4769-9a19-84e9124568a3/volumes"
Oct 31 02:09:46.455103 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:46.454894    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b14aee9c-3186-4e2a-9c84-9b50cb844194" path="/var/lib/kubelet/pods/b14aee9c-3186-4e2a-9c84-9b50cb844194/volumes"
Oct 31 02:09:48.036695 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:48.036667    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:52.710614 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:52.710577    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:52.893556 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:52.893512    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:53.077150 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:53.077119    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:53.750008 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:53.749974    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l" podUID="5efb0ead-026d-472a-b36d-8ad0cccd3487" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:09:54.209659 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:54.209623    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:57.131351 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:57.131311    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:57.482245 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:57.482147    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"]
Oct 31 02:09:57.625754 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:57.625722    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:57.810737 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:57.810701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:58.906618 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:09:58.906560    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:59.182904 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:09:59.182826    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:10:02.375674 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:02.375631    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:02.482980 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:02.482933    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh" podUID="047a012c-e995-469b-a04a-ce12cdd56f94" containerName="payload-pre-processing" containerID="cri-o://8962fcdaac8c054844c17945fa927b24920237fae23e67a8644d2afdf19936f6" gracePeriod=25
Oct 31 02:10:02.678673 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:02.678551    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:03.098708 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:03.098676    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:03.305841 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:03.305810    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:03.750050 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:03.750008    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l" podUID="5efb0ead-026d-472a-b36d-8ad0cccd3487" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:10:04.773555 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:04.773517    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:04.902724 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:04.902688    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"
Oct 31 02:10:05.005156 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.005108    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5efb0ead-026d-472a-b36d-8ad0cccd3487-kube-api-access-h2wpf\" (UniqueName: \"kubernetes.io/projected/5efb0ead-026d-472a-b36d-8ad0cccd3487-kube-api-access-h2wpf\") pod \"5efb0ead-026d-472a-b36d-8ad0cccd3487\" (UID: \"5efb0ead-026d-472a-b36d-8ad0cccd3487\") "
Oct 31 02:10:05.005399 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.005309    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5efb0ead-026d-472a-b36d-8ad0cccd3487-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efb0ead-026d-472a-b36d-8ad0cccd3487-plugins-config-volume\") pod \"5efb0ead-026d-472a-b36d-8ad0cccd3487\" (UID: \"5efb0ead-026d-472a-b36d-8ad0cccd3487\") "
Oct 31 02:10:05.006106 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.006073    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efb0ead-026d-472a-b36d-8ad0cccd3487-plugins-config-volume" pod "5efb0ead-026d-472a-b36d-8ad0cccd3487" (UID: "5efb0ead-026d-472a-b36d-8ad0cccd3487"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:05.006444 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.006426    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5efb0ead-026d-472a-b36d-8ad0cccd3487-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:05.008570 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.008542    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5efb0ead-026d-472a-b36d-8ad0cccd3487-kube-api-access-h2wpf" pod "5efb0ead-026d-472a-b36d-8ad0cccd3487" (UID: "5efb0ead-026d-472a-b36d-8ad0cccd3487"). InnerVolumeSpecName "kube-api-access-h2wpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:05.027345 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.027306    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:05.108058 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.107933    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2wpf\" (UniqueName: \"kubernetes.io/projected/5efb0ead-026d-472a-b36d-8ad0cccd3487-kube-api-access-h2wpf\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:05.428728 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.428609    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:05.589509 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.589478    4476 generic.go:356] "Generic (PLEG): container finished" podID="5efb0ead-026d-472a-b36d-8ad0cccd3487" containerID="705593f2bb7d51b9ea6b91231dc51de412afd5fac74f7d82b86c82f74dbc8217" exitCode=0
Oct 31 02:10:05.589729 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.589554    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"
Oct 31 02:10:05.589729 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.589570    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l" event={"ID":"5efb0ead-026d-472a-b36d-8ad0cccd3487","Type":"ContainerDied","Data":"705593f2bb7d51b9ea6b91231dc51de412afd5fac74f7d82b86c82f74dbc8217"}
Oct 31 02:10:05.589729 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.589637    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l" event={"ID":"5efb0ead-026d-472a-b36d-8ad0cccd3487","Type":"ContainerDied","Data":"aca7e8f4ed873e2296b404a58f237eed13bd6b07a47401ac82b07cc0c02281ea"}
Oct 31 02:10:05.589729 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.589667    4476 scope.go:122] "RemoveContainer" containerID="705593f2bb7d51b9ea6b91231dc51de412afd5fac74f7d82b86c82f74dbc8217"
Oct 31 02:10:05.599470 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.599447    4476 scope.go:122] "RemoveContainer" containerID="705593f2bb7d51b9ea6b91231dc51de412afd5fac74f7d82b86c82f74dbc8217"
Oct 31 02:10:05.599785 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:10:05.599767    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705593f2bb7d51b9ea6b91231dc51de412afd5fac74f7d82b86c82f74dbc8217\": container with ID starting with 705593f2bb7d51b9ea6b91231dc51de412afd5fac74f7d82b86c82f74dbc8217 not found: ID does not exist" containerID="705593f2bb7d51b9ea6b91231dc51de412afd5fac74f7d82b86c82f74dbc8217"
Oct 31 02:10:05.599835 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.599798    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705593f2bb7d51b9ea6b91231dc51de412afd5fac74f7d82b86c82f74dbc8217"} err="failed to get container status \"705593f2bb7d51b9ea6b91231dc51de412afd5fac74f7d82b86c82f74dbc8217\": rpc error: code = NotFound desc = could not find container \"705593f2bb7d51b9ea6b91231dc51de412afd5fac74f7d82b86c82f74dbc8217\": container with ID starting with 705593f2bb7d51b9ea6b91231dc51de412afd5fac74f7d82b86c82f74dbc8217 not found: ID does not exist"
Oct 31 02:10:05.712480 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.712402    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:05.753008 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.752980    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"]
Oct 31 02:10:05.802240 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:05.802212    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7fc058-7d59fcb779-n5j7l"]
Oct 31 02:10:06.454369 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:06.454331    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5efb0ead-026d-472a-b36d-8ad0cccd3487" path="/var/lib/kubelet/pods/5efb0ead-026d-472a-b36d-8ad0cccd3487/volumes"
Oct 31 02:10:08.916753 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:08.916708    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh" podUID="047a012c-e995-469b-a04a-ce12cdd56f94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:10:09.585101 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:09.585064    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:09.792476 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:09.792442    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:09.895842 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:09.895764    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"]
Oct 31 02:10:09.896053 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:09.896031    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt" podUID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerName="istio-proxy" containerID="cri-o://648774f16c689ca7bbaaf98082af6f1567f58502804d935a18daa9270b6fd896" gracePeriod=30
Oct 31 02:10:09.992778 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:09.992749    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:10.198823 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:10.198714    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:10.527843 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:10.527815    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:12.077831 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:12.077794    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:12.319363 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:12.319328    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:12.572717 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:12.572681    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:12.919399 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:12.919314    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:14.622974 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:14.622925    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"]
Oct 31 02:10:14.623452 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:14.623210    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52" podUID="cbfffc20-e86c-42da-b9d2-58d24d9152aa" containerName="maas-api" containerID="cri-o://0ff5a134efcda51384adeaaf97c041a8e4c1eeedf4758586ca77354947dbb863" gracePeriod=30
Oct 31 02:10:14.783682 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:14.783649    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:14.912576 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:14.912543    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 02:10:15.041583 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.041552    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-metrics-certs\") pod \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") "
Oct 31 02:10:15.041780 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.041625    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbfffc20-e86c-42da-b9d2-58d24d9152aa-kube-api-access-qwgkv\" (UniqueName: \"kubernetes.io/projected/cbfffc20-e86c-42da-b9d2-58d24d9152aa-kube-api-access-qwgkv\") pod \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") "
Oct 31 02:10:15.041780 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.041724    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-maas-api-tls\") pod \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") "
Oct 31 02:10:15.041780 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.041753    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbfffc20-e86c-42da-b9d2-58d24d9152aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfffc20-e86c-42da-b9d2-58d24d9152aa-openshift-service-ca\") pod \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\" (UID: \"cbfffc20-e86c-42da-b9d2-58d24d9152aa\") "
Oct 31 02:10:15.042557 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.042386    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbfffc20-e86c-42da-b9d2-58d24d9152aa-openshift-service-ca" pod "cbfffc20-e86c-42da-b9d2-58d24d9152aa" (UID: "cbfffc20-e86c-42da-b9d2-58d24d9152aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:15.044306 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.044277    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-metrics-certs" pod "cbfffc20-e86c-42da-b9d2-58d24d9152aa" (UID: "cbfffc20-e86c-42da-b9d2-58d24d9152aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:15.044444 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.044407    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfffc20-e86c-42da-b9d2-58d24d9152aa-kube-api-access-qwgkv" pod "cbfffc20-e86c-42da-b9d2-58d24d9152aa" (UID: "cbfffc20-e86c-42da-b9d2-58d24d9152aa"). InnerVolumeSpecName "kube-api-access-qwgkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:15.045331 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.045302    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-maas-api-tls" pod "cbfffc20-e86c-42da-b9d2-58d24d9152aa" (UID: "cbfffc20-e86c-42da-b9d2-58d24d9152aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:15.145153 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.145035    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:15.145153 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.145078    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwgkv\" (UniqueName: \"kubernetes.io/projected/cbfffc20-e86c-42da-b9d2-58d24d9152aa-kube-api-access-qwgkv\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:15.145153 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.145093    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbfffc20-e86c-42da-b9d2-58d24d9152aa-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:15.145153 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.145106    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfffc20-e86c-42da-b9d2-58d24d9152aa-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:15.164390 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.164362    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 02:10:15.246057 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246026    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-workload-socket\") pod \"069276a8-4b8e-4db0-9dd1-86c7b402309d\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") "
Oct 31 02:10:15.246248 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246119    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069276a8-4b8e-4db0-9dd1-86c7b402309d-kube-api-access-kxbb5\" (UniqueName: \"kubernetes.io/projected/069276a8-4b8e-4db0-9dd1-86c7b402309d-kube-api-access-kxbb5\") pod \"069276a8-4b8e-4db0-9dd1-86c7b402309d\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") "
Oct 31 02:10:15.246248 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246219    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-envoy\") pod \"069276a8-4b8e-4db0-9dd1-86c7b402309d\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") "
Oct 31 02:10:15.246475 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246265    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-podinfo\") pod \"069276a8-4b8e-4db0-9dd1-86c7b402309d\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") "
Oct 31 02:10:15.246475 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246311    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-credential-socket\") pod \"069276a8-4b8e-4db0-9dd1-86c7b402309d\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") "
Oct 31 02:10:15.246475 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246341    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-data\") pod \"069276a8-4b8e-4db0-9dd1-86c7b402309d\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") "
Oct 31 02:10:15.246475 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246370    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/069276a8-4b8e-4db0-9dd1-86c7b402309d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069276a8-4b8e-4db0-9dd1-86c7b402309d-istiod-ca-cert\") pod \"069276a8-4b8e-4db0-9dd1-86c7b402309d\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") "
Oct 31 02:10:15.246475 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246399    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-token\" (UniqueName: \"kubernetes.io/projected/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-token\") pod \"069276a8-4b8e-4db0-9dd1-86c7b402309d\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") "
Oct 31 02:10:15.246475 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246427    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-workload-certs\") pod \"069276a8-4b8e-4db0-9dd1-86c7b402309d\" (UID: \"069276a8-4b8e-4db0-9dd1-86c7b402309d\") "
Oct 31 02:10:15.246817 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246380    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-workload-socket" pod "069276a8-4b8e-4db0-9dd1-86c7b402309d" (UID: "069276a8-4b8e-4db0-9dd1-86c7b402309d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:15.246879 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246612    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-credential-socket" pod "069276a8-4b8e-4db0-9dd1-86c7b402309d" (UID: "069276a8-4b8e-4db0-9dd1-86c7b402309d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:15.246879 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246628    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-data" pod "069276a8-4b8e-4db0-9dd1-86c7b402309d" (UID: "069276a8-4b8e-4db0-9dd1-86c7b402309d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:15.246879 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246766    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069276a8-4b8e-4db0-9dd1-86c7b402309d-istiod-ca-cert" pod "069276a8-4b8e-4db0-9dd1-86c7b402309d" (UID: "069276a8-4b8e-4db0-9dd1-86c7b402309d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:15.247023 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246962    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:15.247023 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246983    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:15.247109 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246942    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-workload-certs" pod "069276a8-4b8e-4db0-9dd1-86c7b402309d" (UID: "069276a8-4b8e-4db0-9dd1-86c7b402309d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:15.247109 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.246998    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069276a8-4b8e-4db0-9dd1-86c7b402309d-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:15.247208 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.247120    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:15.248583 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.248562    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069276a8-4b8e-4db0-9dd1-86c7b402309d-kube-api-access-kxbb5" pod "069276a8-4b8e-4db0-9dd1-86c7b402309d" (UID: "069276a8-4b8e-4db0-9dd1-86c7b402309d"). InnerVolumeSpecName "kube-api-access-kxbb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:15.248815 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.248797    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-token" pod "069276a8-4b8e-4db0-9dd1-86c7b402309d" (UID: "069276a8-4b8e-4db0-9dd1-86c7b402309d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:15.249110 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.249083    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-envoy" pod "069276a8-4b8e-4db0-9dd1-86c7b402309d" (UID: "069276a8-4b8e-4db0-9dd1-86c7b402309d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:15.249110 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.249099    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-podinfo" pod "069276a8-4b8e-4db0-9dd1-86c7b402309d" (UID: "069276a8-4b8e-4db0-9dd1-86c7b402309d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:10:15.347989 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.347953    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxbb5\" (UniqueName: \"kubernetes.io/projected/069276a8-4b8e-4db0-9dd1-86c7b402309d-kube-api-access-kxbb5\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:15.347989 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.347985    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:15.347989 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.347994    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:15.348264 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.348003    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069276a8-4b8e-4db0-9dd1-86c7b402309d-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:15.348264 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.348011    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069276a8-4b8e-4db0-9dd1-86c7b402309d-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:15.598348 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.598314    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:15.648812 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.648782    4476 generic.go:356] "Generic (PLEG): container finished" podID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerID="648774f16c689ca7bbaaf98082af6f1567f58502804d935a18daa9270b6fd896" exitCode=0
Oct 31 02:10:15.649283 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.648841    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"
Oct 31 02:10:15.649283 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.648867    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt" event={"ID":"069276a8-4b8e-4db0-9dd1-86c7b402309d","Type":"ContainerDied","Data":"648774f16c689ca7bbaaf98082af6f1567f58502804d935a18daa9270b6fd896"}
Oct 31 02:10:15.649283 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.648929    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt" event={"ID":"069276a8-4b8e-4db0-9dd1-86c7b402309d","Type":"ContainerDied","Data":"f16eb21e971a32c55f916639c1cd9869a3ff707229b0dcc9c3a55e3c95a34f81"}
Oct 31 02:10:15.649283 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.648950    4476 scope.go:122] "RemoveContainer" containerID="648774f16c689ca7bbaaf98082af6f1567f58502804d935a18daa9270b6fd896"
Oct 31 02:10:15.650420 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.650370    4476 generic.go:356] "Generic (PLEG): container finished" podID="cbfffc20-e86c-42da-b9d2-58d24d9152aa" containerID="0ff5a134efcda51384adeaaf97c041a8e4c1eeedf4758586ca77354947dbb863" exitCode=0
Oct 31 02:10:15.650503 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.650419    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52" event={"ID":"cbfffc20-e86c-42da-b9d2-58d24d9152aa","Type":"ContainerDied","Data":"0ff5a134efcda51384adeaaf97c041a8e4c1eeedf4758586ca77354947dbb863"}
Oct 31 02:10:15.650503 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.650431    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"
Oct 31 02:10:15.650503 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.650464    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52" event={"ID":"cbfffc20-e86c-42da-b9d2-58d24d9152aa","Type":"ContainerDied","Data":"87f83e7867796a037702332739aa34bd460fc1ec44ce4009e8cccdada6803855"}
Oct 31 02:10:15.659257 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.659237    4476 scope.go:122] "RemoveContainer" containerID="fb6ebf7571a9c70935bbe989502cc41023bd102e899e9361385a4fe02dae4de3"
Oct 31 02:10:15.668732 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.668714    4476 scope.go:122] "RemoveContainer" containerID="648774f16c689ca7bbaaf98082af6f1567f58502804d935a18daa9270b6fd896"
Oct 31 02:10:15.669038 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:10:15.669013    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648774f16c689ca7bbaaf98082af6f1567f58502804d935a18daa9270b6fd896\": container with ID starting with 648774f16c689ca7bbaaf98082af6f1567f58502804d935a18daa9270b6fd896 not found: ID does not exist" containerID="648774f16c689ca7bbaaf98082af6f1567f58502804d935a18daa9270b6fd896"
Oct 31 02:10:15.669103 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.669050    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648774f16c689ca7bbaaf98082af6f1567f58502804d935a18daa9270b6fd896"} err="failed to get container status \"648774f16c689ca7bbaaf98082af6f1567f58502804d935a18daa9270b6fd896\": rpc error: code = NotFound desc = could not find container \"648774f16c689ca7bbaaf98082af6f1567f58502804d935a18daa9270b6fd896\": container with ID starting with 648774f16c689ca7bbaaf98082af6f1567f58502804d935a18daa9270b6fd896 not found: ID does not exist"
Oct 31 02:10:15.669103 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.669076    4476 scope.go:122] "RemoveContainer" containerID="fb6ebf7571a9c70935bbe989502cc41023bd102e899e9361385a4fe02dae4de3"
Oct 31 02:10:15.669385 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:10:15.669368    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6ebf7571a9c70935bbe989502cc41023bd102e899e9361385a4fe02dae4de3\": container with ID starting with fb6ebf7571a9c70935bbe989502cc41023bd102e899e9361385a4fe02dae4de3 not found: ID does not exist" containerID="fb6ebf7571a9c70935bbe989502cc41023bd102e899e9361385a4fe02dae4de3"
Oct 31 02:10:15.669456 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.669391    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6ebf7571a9c70935bbe989502cc41023bd102e899e9361385a4fe02dae4de3"} err="failed to get container status \"fb6ebf7571a9c70935bbe989502cc41023bd102e899e9361385a4fe02dae4de3\": rpc error: code = NotFound desc = could not find container \"fb6ebf7571a9c70935bbe989502cc41023bd102e899e9361385a4fe02dae4de3\": container with ID starting with fb6ebf7571a9c70935bbe989502cc41023bd102e899e9361385a4fe02dae4de3 not found: ID does not exist"
Oct 31 02:10:15.669456 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.669408    4476 scope.go:122] "RemoveContainer" containerID="0ff5a134efcda51384adeaaf97c041a8e4c1eeedf4758586ca77354947dbb863"
Oct 31 02:10:15.680452 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.680430    4476 scope.go:122] "RemoveContainer" containerID="0ff5a134efcda51384adeaaf97c041a8e4c1eeedf4758586ca77354947dbb863"
Oct 31 02:10:15.680745 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:10:15.680720    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff5a134efcda51384adeaaf97c041a8e4c1eeedf4758586ca77354947dbb863\": container with ID starting with 0ff5a134efcda51384adeaaf97c041a8e4c1eeedf4758586ca77354947dbb863 not found: ID does not exist" containerID="0ff5a134efcda51384adeaaf97c041a8e4c1eeedf4758586ca77354947dbb863"
Oct 31 02:10:15.680866 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.680753    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff5a134efcda51384adeaaf97c041a8e4c1eeedf4758586ca77354947dbb863"} err="failed to get container status \"0ff5a134efcda51384adeaaf97c041a8e4c1eeedf4758586ca77354947dbb863\": rpc error: code = NotFound desc = could not find container \"0ff5a134efcda51384adeaaf97c041a8e4c1eeedf4758586ca77354947dbb863\": container with ID starting with 0ff5a134efcda51384adeaaf97c041a8e4c1eeedf4758586ca77354947dbb863 not found: ID does not exist"
Oct 31 02:10:15.812154 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.812105    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"]
Oct 31 02:10:15.861947 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.861850    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c9c3c-openshift-default-54f8bc8bf7-c2dxt"]
Oct 31 02:10:15.926293 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.926255    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"]
Oct 31 02:10:15.929368 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:15.929342    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7fc058-6c5dd47745-sdk52"]
Oct 31 02:10:16.393864 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:16.393827    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:16.453569 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:16.453518    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="069276a8-4b8e-4db0-9dd1-86c7b402309d" path="/var/lib/kubelet/pods/069276a8-4b8e-4db0-9dd1-86c7b402309d/volumes"
Oct 31 02:10:16.454079 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:16.454059    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbfffc20-e86c-42da-b9d2-58d24d9152aa" path="/var/lib/kubelet/pods/cbfffc20-e86c-42da-b9d2-58d24d9152aa/volumes"
Oct 31 02:10:16.658063 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:16.657983    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:16.847816 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:16.847780    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:17.346979 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:17.346945    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:17.824933 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:17.824905    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"]
Oct 31 02:10:17.874279 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:17.874247    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"]
Oct 31 02:10:17.876588 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:17.876561    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"]
Oct 31 02:10:17.977057 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:17.977024    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:18.917240 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:18.917155    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh" podUID="047a012c-e995-469b-a04a-ce12cdd56f94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:10:21.782788 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:21.782685    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:22.393270 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:22.393234    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"]
Oct 31 02:10:22.531799 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:22.531766    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"]
Oct 31 02:10:22.696723 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:22.696684    4476 generic.go:356] "Generic (PLEG): container finished" podID="047a012c-e995-469b-a04a-ce12cdd56f94" containerID="8962fcdaac8c054844c17945fa927b24920237fae23e67a8644d2afdf19936f6" exitCode=0
Oct 31 02:10:22.696895 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:22.696751    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh" event={"ID":"047a012c-e995-469b-a04a-ce12cdd56f94","Type":"ContainerDied","Data":"8962fcdaac8c054844c17945fa927b24920237fae23e67a8644d2afdf19936f6"}
Oct 31 02:10:22.751872 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:22.751842    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"
Oct 31 02:10:22.823110 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:22.823079    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:22.858231 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:22.846608    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/047a012c-e995-469b-a04a-ce12cdd56f94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047a012c-e995-469b-a04a-ce12cdd56f94-plugins-config-volume\") pod \"047a012c-e995-469b-a04a-ce12cdd56f94\" (UID: \"047a012c-e995-469b-a04a-ce12cdd56f94\") "
Oct 31 02:10:22.858231 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:22.846991    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047a012c-e995-469b-a04a-ce12cdd56f94-plugins-config-volume" pod "047a012c-e995-469b-a04a-ce12cdd56f94" (UID: "047a012c-e995-469b-a04a-ce12cdd56f94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:22.858231 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:22.847026    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047a012c-e995-469b-a04a-ce12cdd56f94-kube-api-access-vvmcx\" (UniqueName: \"kubernetes.io/projected/047a012c-e995-469b-a04a-ce12cdd56f94-kube-api-access-vvmcx\") pod \"047a012c-e995-469b-a04a-ce12cdd56f94\" (UID: \"047a012c-e995-469b-a04a-ce12cdd56f94\") "
Oct 31 02:10:22.858231 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:22.847480    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/047a012c-e995-469b-a04a-ce12cdd56f94-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:22.858231 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:22.853384    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047a012c-e995-469b-a04a-ce12cdd56f94-kube-api-access-vvmcx" pod "047a012c-e995-469b-a04a-ce12cdd56f94" (UID: "047a012c-e995-469b-a04a-ce12cdd56f94"). InnerVolumeSpecName "kube-api-access-vvmcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:22.948677 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:22.948588    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvmcx\" (UniqueName: \"kubernetes.io/projected/047a012c-e995-469b-a04a-ce12cdd56f94-kube-api-access-vvmcx\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:23.012521 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:23.012483    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:23.510392 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:23.510324    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:23.691858 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:23.691800    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:23.703983 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:23.703959    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"
Oct 31 02:10:23.704120 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:23.704000    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh" event={"ID":"047a012c-e995-469b-a04a-ce12cdd56f94","Type":"ContainerDied","Data":"8c2babfca644daa7d7b0f1817a178166050d6fe6cd8eb55b28dd53ff3eb45af4"}
Oct 31 02:10:23.704120 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:23.704053    4476 scope.go:122] "RemoveContainer" containerID="8962fcdaac8c054844c17945fa927b24920237fae23e67a8644d2afdf19936f6"
Oct 31 02:10:23.868407 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:23.868374    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"]
Oct 31 02:10:23.883445 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:23.883404    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:23.917612 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:23.917576    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-9c9c3c-7dd967ff87-pbrqh"]
Oct 31 02:10:24.328817 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:24.328778    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:24.454339 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:24.454303    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="047a012c-e995-469b-a04a-ce12cdd56f94" path="/var/lib/kubelet/pods/047a012c-e995-469b-a04a-ce12cdd56f94/volumes"
Oct 31 02:10:26.506879 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:26.506843    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:26.686344 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:26.686308    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:26.861935 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:26.861905    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:27.393822 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.393773    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn" podUID="a7b29e19-6e82-4c47-a9f6-d66f60afd3ca" containerName="payload-processing" containerID="cri-o://c246c9b74ff834a5cf1711b363d84d70425cc6c1b7c0a316a249d4c7efd8a808" gracePeriod=25
Oct 31 02:10:27.532852 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.532816    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct" podUID="2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912" containerName="payload-pre-processing" containerID="cri-o://874e7836b191eeb22b86ad6b2d42ddab91f5491b571f438190da765ff095ec7e" gracePeriod=25
Oct 31 02:10:27.661933 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.661907    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"
Oct 31 02:10:27.728570 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.728532    4476 generic.go:356] "Generic (PLEG): container finished" podID="a7b29e19-6e82-4c47-a9f6-d66f60afd3ca" containerID="c246c9b74ff834a5cf1711b363d84d70425cc6c1b7c0a316a249d4c7efd8a808" exitCode=0
Oct 31 02:10:27.728813 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.728598    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"
Oct 31 02:10:27.728813 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.728624    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn" event={"ID":"a7b29e19-6e82-4c47-a9f6-d66f60afd3ca","Type":"ContainerDied","Data":"c246c9b74ff834a5cf1711b363d84d70425cc6c1b7c0a316a249d4c7efd8a808"}
Oct 31 02:10:27.728813 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.728730    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn" event={"ID":"a7b29e19-6e82-4c47-a9f6-d66f60afd3ca","Type":"ContainerDied","Data":"37478683f9fcdb1e2bea46d6f82f4ade89adade9d476c2cc855f3310451558e0"}
Oct 31 02:10:27.728813 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.728803    4476 scope.go:122] "RemoveContainer" containerID="c246c9b74ff834a5cf1711b363d84d70425cc6c1b7c0a316a249d4c7efd8a808"
Oct 31 02:10:27.735410 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.732586    4476 generic.go:356] "Generic (PLEG): container finished" podID="2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912" containerID="874e7836b191eeb22b86ad6b2d42ddab91f5491b571f438190da765ff095ec7e" exitCode=0
Oct 31 02:10:27.735410 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.733116    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct" event={"ID":"2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912","Type":"ContainerDied","Data":"874e7836b191eeb22b86ad6b2d42ddab91f5491b571f438190da765ff095ec7e"}
Oct 31 02:10:27.746692 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.742729    4476 scope.go:122] "RemoveContainer" containerID="c246c9b74ff834a5cf1711b363d84d70425cc6c1b7c0a316a249d4c7efd8a808"
Oct 31 02:10:27.746692 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:10:27.743058    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c246c9b74ff834a5cf1711b363d84d70425cc6c1b7c0a316a249d4c7efd8a808\": container with ID starting with c246c9b74ff834a5cf1711b363d84d70425cc6c1b7c0a316a249d4c7efd8a808 not found: ID does not exist" containerID="c246c9b74ff834a5cf1711b363d84d70425cc6c1b7c0a316a249d4c7efd8a808"
Oct 31 02:10:27.746692 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.743090    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c246c9b74ff834a5cf1711b363d84d70425cc6c1b7c0a316a249d4c7efd8a808"} err="failed to get container status \"c246c9b74ff834a5cf1711b363d84d70425cc6c1b7c0a316a249d4c7efd8a808\": rpc error: code = NotFound desc = could not find container \"c246c9b74ff834a5cf1711b363d84d70425cc6c1b7c0a316a249d4c7efd8a808\": container with ID starting with c246c9b74ff834a5cf1711b363d84d70425cc6c1b7c0a316a249d4c7efd8a808 not found: ID does not exist"
Oct 31 02:10:27.798029 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.798005    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"
Oct 31 02:10:27.802104 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.802084    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca-kube-api-access-vglcd\" (UniqueName: \"kubernetes.io/projected/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca-kube-api-access-vglcd\") pod \"a7b29e19-6e82-4c47-a9f6-d66f60afd3ca\" (UID: \"a7b29e19-6e82-4c47-a9f6-d66f60afd3ca\") "
Oct 31 02:10:27.802294 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.802278    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca-plugins-config-volume\") pod \"a7b29e19-6e82-4c47-a9f6-d66f60afd3ca\" (UID: \"a7b29e19-6e82-4c47-a9f6-d66f60afd3ca\") "
Oct 31 02:10:27.802648 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.802601    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca-plugins-config-volume" pod "a7b29e19-6e82-4c47-a9f6-d66f60afd3ca" (UID: "a7b29e19-6e82-4c47-a9f6-d66f60afd3ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:27.804554 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.804530    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca-kube-api-access-vglcd" pod "a7b29e19-6e82-4c47-a9f6-d66f60afd3ca" (UID: "a7b29e19-6e82-4c47-a9f6-d66f60afd3ca"). InnerVolumeSpecName "kube-api-access-vglcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:27.902942 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.902862    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912-plugins-config-volume\") pod \"2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912\" (UID: \"2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912\") "
Oct 31 02:10:27.903117 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.903100    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912-kube-api-access-8r2m2\" (UniqueName: \"kubernetes.io/projected/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912-kube-api-access-8r2m2\") pod \"2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912\" (UID: \"2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912\") "
Oct 31 02:10:27.903331 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.903311    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912-plugins-config-volume" pod "2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912" (UID: "2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:27.903550 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.903526    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:27.903550 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.903549    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:27.903703 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.903565    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vglcd\" (UniqueName: \"kubernetes.io/projected/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca-kube-api-access-vglcd\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:27.905303 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:27.905280    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912-kube-api-access-8r2m2" pod "2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912" (UID: "2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912"). InnerVolumeSpecName "kube-api-access-8r2m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:28.004997 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:28.004962    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r2m2\" (UniqueName: \"kubernetes.io/projected/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912-kube-api-access-8r2m2\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:28.151161 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:28.151128    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"]
Oct 31 02:10:28.154934 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:28.154870    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e621db-68d4bc4bc5-hrkvn"]
Oct 31 02:10:28.453599 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:28.453569    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7b29e19-6e82-4c47-a9f6-d66f60afd3ca" path="/var/lib/kubelet/pods/a7b29e19-6e82-4c47-a9f6-d66f60afd3ca/volumes"
Oct 31 02:10:28.739715 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:28.739623    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"
Oct 31 02:10:28.739715 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:28.739636    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct" event={"ID":"2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912","Type":"ContainerDied","Data":"ee056a32cfc66d9eec37f7af2669c909cd77988ae6bffe7c43060bd944b2f7c2"}
Oct 31 02:10:28.739715 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:28.739698    4476 scope.go:122] "RemoveContainer" containerID="874e7836b191eeb22b86ad6b2d42ddab91f5491b571f438190da765ff095ec7e"
Oct 31 02:10:28.902697 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:28.902654    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"]
Oct 31 02:10:28.908499 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:28.908471    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e621db-55897f959-crnct"]
Oct 31 02:10:29.182124 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:10:29.182080    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:10:29.218792 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:29.218763    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:29.235539 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:29.235510    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"]
Oct 31 02:10:29.240961 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:29.240938    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"]
Oct 31 02:10:29.431662 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:29.431633    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:30.453746 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:30.453714    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912" path="/var/lib/kubelet/pods/2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912/volumes"
Oct 31 02:10:31.931478 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:31.931388    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:32.845935 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:32.845896    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4" podUID="762230e3-0692-4a5b-8e29-13bf8a3a4b3c" containerName="main" containerID="cri-o://e404de57619084613d3f871e3b5a8ecabe5bc8d81c7a3864f18a5a3726bfd27b" gracePeriod=45
Oct 31 02:10:32.894486 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:32.894432    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" podUID="27377642-d40d-4498-af5d-16ca6a58ef54" containerName="main" containerID="cri-o://51676a73d4dd485f021f44784500662f8c1074b6baefd3724687ea1d6b73b65e" gracePeriod=45
Oct 31 02:10:32.895982 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:32.895954    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerName="main" containerID="cri-o://ae972356d26187494eb58316cd76e3d9dafd9e1ac9fded1e3bc3940e89788eca" gracePeriod=45
Oct 31 02:10:32.904905 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:10:32.904879    4476 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod762230e3_0692_4a5b_8e29_13bf8a3a4b3c.slice/crio-conmon-e404de57619084613d3f871e3b5a8ecabe5bc8d81c7a3864f18a5a3726bfd27b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 02:10:33.173536 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.173505    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 02:10:33.193473 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.193448    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 02:10:33.205395 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.205373    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 02:10:33.259707 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.259621    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-kube-api-access-xv5xl\" (UniqueName: \"kubernetes.io/projected/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-kube-api-access-xv5xl\") pod \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") "
Oct 31 02:10:33.259844 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.259741    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-model-cache\") pod \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") "
Oct 31 02:10:33.259844 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.259766    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-dshm\") pod \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") "
Oct 31 02:10:33.259934 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.259908    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-home\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-home\") pod \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") "
Oct 31 02:10:33.259983 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.259950    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-tmp-dir\") pod \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") "
Oct 31 02:10:33.259983 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.259966    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-tls-certs\" (UniqueName: \"kubernetes.io/secret/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-tls-certs\") pod \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\" (UID: \"762230e3-0692-4a5b-8e29-13bf8a3a4b3c\") "
Oct 31 02:10:33.260086 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.260021    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-model-cache" pod "762230e3-0692-4a5b-8e29-13bf8a3a4b3c" (UID: "762230e3-0692-4a5b-8e29-13bf8a3a4b3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:33.260272 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.260249    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-home" pod "762230e3-0692-4a5b-8e29-13bf8a3a4b3c" (UID: "762230e3-0692-4a5b-8e29-13bf8a3a4b3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:33.260373 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.260265    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-tmp-dir" pod "762230e3-0692-4a5b-8e29-13bf8a3a4b3c" (UID: "762230e3-0692-4a5b-8e29-13bf8a3a4b3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:33.260434 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.260399    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.260434 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.260419    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.260512 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.260436    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.262043 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.262018    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-dshm" pod "762230e3-0692-4a5b-8e29-13bf8a3a4b3c" (UID: "762230e3-0692-4a5b-8e29-13bf8a3a4b3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:33.262043 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.262032    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-kube-api-access-xv5xl" pod "762230e3-0692-4a5b-8e29-13bf8a3a4b3c" (UID: "762230e3-0692-4a5b-8e29-13bf8a3a4b3c"). InnerVolumeSpecName "kube-api-access-xv5xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:33.262248 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.262232    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-tls-certs" pod "762230e3-0692-4a5b-8e29-13bf8a3a4b3c" (UID: "762230e3-0692-4a5b-8e29-13bf8a3a4b3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:33.361476 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.361445    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbdc50dd-29d0-4610-a361-cfd11ad1f501-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbdc50dd-29d0-4610-a361-cfd11ad1f501-tls-certs\") pod \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") "
Oct 31 02:10:33.361650 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.361488    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-dshm\") pod \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") "
Oct 31 02:10:33.361650 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.361543    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdc50dd-29d0-4610-a361-cfd11ad1f501-kube-api-access-8wkxm\" (UniqueName: \"kubernetes.io/projected/cbdc50dd-29d0-4610-a361-cfd11ad1f501-kube-api-access-8wkxm\") pod \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") "
Oct 31 02:10:33.361650 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.361595    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-tmp-dir\") pod \"27377642-d40d-4498-af5d-16ca6a58ef54\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") "
Oct 31 02:10:33.361650 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.361633    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-dshm\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-dshm\") pod \"27377642-d40d-4498-af5d-16ca6a58ef54\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") "
Oct 31 02:10:33.361952 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.361675    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27377642-d40d-4498-af5d-16ca6a58ef54-tls-certs\" (UniqueName: \"kubernetes.io/secret/27377642-d40d-4498-af5d-16ca6a58ef54-tls-certs\") pod \"27377642-d40d-4498-af5d-16ca6a58ef54\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") "
Oct 31 02:10:33.361952 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.361734    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-tmp-dir\") pod \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") "
Oct 31 02:10:33.361952 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.361785    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-home\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-home\") pod \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") "
Oct 31 02:10:33.361952 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.361829    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-model-cache\") pod \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\" (UID: \"cbdc50dd-29d0-4610-a361-cfd11ad1f501\") "
Oct 31 02:10:33.361952 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.361859    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-tmp-dir" pod "27377642-d40d-4498-af5d-16ca6a58ef54" (UID: "27377642-d40d-4498-af5d-16ca6a58ef54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:33.361952 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.361886    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-home\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-home\") pod \"27377642-d40d-4498-af5d-16ca6a58ef54\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") "
Oct 31 02:10:33.361952 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.361923    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27377642-d40d-4498-af5d-16ca6a58ef54-kube-api-access-2xkpm\" (UniqueName: \"kubernetes.io/projected/27377642-d40d-4498-af5d-16ca6a58ef54-kube-api-access-2xkpm\") pod \"27377642-d40d-4498-af5d-16ca6a58ef54\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") "
Oct 31 02:10:33.362339 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.361960    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-model-cache\") pod \"27377642-d40d-4498-af5d-16ca6a58ef54\" (UID: \"27377642-d40d-4498-af5d-16ca6a58ef54\") "
Oct 31 02:10:33.362339 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.362086    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-home" pod "cbdc50dd-29d0-4610-a361-cfd11ad1f501" (UID: "cbdc50dd-29d0-4610-a361-cfd11ad1f501"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:33.362339 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.362281    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-home" pod "27377642-d40d-4498-af5d-16ca6a58ef54" (UID: "27377642-d40d-4498-af5d-16ca6a58ef54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:33.362618 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.362595    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-tmp-dir" pod "cbdc50dd-29d0-4610-a361-cfd11ad1f501" (UID: "cbdc50dd-29d0-4610-a361-cfd11ad1f501"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:33.362741 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.362672    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.362741 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.362677    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-model-cache" pod "cbdc50dd-29d0-4610-a361-cfd11ad1f501" (UID: "cbdc50dd-29d0-4610-a361-cfd11ad1f501"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:33.362741 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.362696    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.362741 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.362713    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv5xl\" (UniqueName: \"kubernetes.io/projected/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-kube-api-access-xv5xl\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.362741 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.362728    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.362741 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.362741    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762230e3-0692-4a5b-8e29-13bf8a3a4b3c-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.362741 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.362755    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.363625 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.362779    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-model-cache" pod "27377642-d40d-4498-af5d-16ca6a58ef54" (UID: "27377642-d40d-4498-af5d-16ca6a58ef54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:33.364591 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.364563    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbdc50dd-29d0-4610-a361-cfd11ad1f501-tls-certs" pod "cbdc50dd-29d0-4610-a361-cfd11ad1f501" (UID: "cbdc50dd-29d0-4610-a361-cfd11ad1f501"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:33.364591 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.364587    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27377642-d40d-4498-af5d-16ca6a58ef54-tls-certs" pod "27377642-d40d-4498-af5d-16ca6a58ef54" (UID: "27377642-d40d-4498-af5d-16ca6a58ef54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:33.364839 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.364809    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdc50dd-29d0-4610-a361-cfd11ad1f501-kube-api-access-8wkxm" pod "cbdc50dd-29d0-4610-a361-cfd11ad1f501" (UID: "cbdc50dd-29d0-4610-a361-cfd11ad1f501"). InnerVolumeSpecName "kube-api-access-8wkxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:33.364928 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.364840    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27377642-d40d-4498-af5d-16ca6a58ef54-kube-api-access-2xkpm" pod "27377642-d40d-4498-af5d-16ca6a58ef54" (UID: "27377642-d40d-4498-af5d-16ca6a58ef54"). InnerVolumeSpecName "kube-api-access-2xkpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:33.364975 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.364938    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-dshm" pod "27377642-d40d-4498-af5d-16ca6a58ef54" (UID: "27377642-d40d-4498-af5d-16ca6a58ef54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:33.365509 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.365491    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-dshm" pod "cbdc50dd-29d0-4610-a361-cfd11ad1f501" (UID: "cbdc50dd-29d0-4610-a361-cfd11ad1f501"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:33.463482 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.463400    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xkpm\" (UniqueName: \"kubernetes.io/projected/27377642-d40d-4498-af5d-16ca6a58ef54-kube-api-access-2xkpm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.463482 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.463438    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.463482 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.463451    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbdc50dd-29d0-4610-a361-cfd11ad1f501-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.463482 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.463463    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.463482 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.463475    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wkxm\" (UniqueName: \"kubernetes.io/projected/cbdc50dd-29d0-4610-a361-cfd11ad1f501-kube-api-access-8wkxm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.463768 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.463496    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27377642-d40d-4498-af5d-16ca6a58ef54-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.463768 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.463508    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27377642-d40d-4498-af5d-16ca6a58ef54-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.463768 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.463519    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.463768 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.463531    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbdc50dd-29d0-4610-a361-cfd11ad1f501-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:33.779006 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.778971    4476 generic.go:356] "Generic (PLEG): container finished" podID="762230e3-0692-4a5b-8e29-13bf8a3a4b3c" containerID="e404de57619084613d3f871e3b5a8ecabe5bc8d81c7a3864f18a5a3726bfd27b" exitCode=0
Oct 31 02:10:33.779315 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.779056    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"
Oct 31 02:10:33.779315 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.779078    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4" event={"ID":"762230e3-0692-4a5b-8e29-13bf8a3a4b3c","Type":"ContainerDied","Data":"e404de57619084613d3f871e3b5a8ecabe5bc8d81c7a3864f18a5a3726bfd27b"}
Oct 31 02:10:33.779315 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.779126    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4" event={"ID":"762230e3-0692-4a5b-8e29-13bf8a3a4b3c","Type":"ContainerDied","Data":"9fec7f80e694370f7989d5f4e914527ee02da719e4f4fea7b6fcdfe4e9f54018"}
Oct 31 02:10:33.779315 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.779144    4476 scope.go:122] "RemoveContainer" containerID="e404de57619084613d3f871e3b5a8ecabe5bc8d81c7a3864f18a5a3726bfd27b"
Oct 31 02:10:33.780504 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.780453    4476 generic.go:356] "Generic (PLEG): container finished" podID="27377642-d40d-4498-af5d-16ca6a58ef54" containerID="51676a73d4dd485f021f44784500662f8c1074b6baefd3724687ea1d6b73b65e" exitCode=0
Oct 31 02:10:33.780605 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.780509    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"
Oct 31 02:10:33.780605 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.780529    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" event={"ID":"27377642-d40d-4498-af5d-16ca6a58ef54","Type":"ContainerDied","Data":"51676a73d4dd485f021f44784500662f8c1074b6baefd3724687ea1d6b73b65e"}
Oct 31 02:10:33.780605 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.780571    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28" event={"ID":"27377642-d40d-4498-af5d-16ca6a58ef54","Type":"ContainerDied","Data":"15f7546643e606384cfe38d10cf63e72bcbe062dfb5ac8a5d48173f1f3173dca"}
Oct 31 02:10:33.782021 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.781998    4476 generic.go:356] "Generic (PLEG): container finished" podID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerID="ae972356d26187494eb58316cd76e3d9dafd9e1ac9fded1e3bc3940e89788eca" exitCode=0
Oct 31 02:10:33.782158 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.782059    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"
Oct 31 02:10:33.782158 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.782114    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" event={"ID":"cbdc50dd-29d0-4610-a361-cfd11ad1f501","Type":"ContainerDied","Data":"ae972356d26187494eb58316cd76e3d9dafd9e1ac9fded1e3bc3940e89788eca"}
Oct 31 02:10:33.782158 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.782141    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk" event={"ID":"cbdc50dd-29d0-4610-a361-cfd11ad1f501","Type":"ContainerDied","Data":"ddffe24a5278a90dfa87103140aead9d0f24acd11c2a0c7cf1d009cc4a3fe5b6"}
Oct 31 02:10:33.789859 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.789840    4476 scope.go:122] "RemoveContainer" containerID="e404de57619084613d3f871e3b5a8ecabe5bc8d81c7a3864f18a5a3726bfd27b"
Oct 31 02:10:33.790362 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:10:33.790113    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e404de57619084613d3f871e3b5a8ecabe5bc8d81c7a3864f18a5a3726bfd27b\": container with ID starting with e404de57619084613d3f871e3b5a8ecabe5bc8d81c7a3864f18a5a3726bfd27b not found: ID does not exist" containerID="e404de57619084613d3f871e3b5a8ecabe5bc8d81c7a3864f18a5a3726bfd27b"
Oct 31 02:10:33.790362 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.790143    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e404de57619084613d3f871e3b5a8ecabe5bc8d81c7a3864f18a5a3726bfd27b"} err="failed to get container status \"e404de57619084613d3f871e3b5a8ecabe5bc8d81c7a3864f18a5a3726bfd27b\": rpc error: code = NotFound desc = could not find container \"e404de57619084613d3f871e3b5a8ecabe5bc8d81c7a3864f18a5a3726bfd27b\": container with ID starting with e404de57619084613d3f871e3b5a8ecabe5bc8d81c7a3864f18a5a3726bfd27b not found: ID does not exist"
Oct 31 02:10:33.790362 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.790168    4476 scope.go:122] "RemoveContainer" containerID="51676a73d4dd485f021f44784500662f8c1074b6baefd3724687ea1d6b73b65e"
Oct 31 02:10:33.798519 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.798500    4476 scope.go:122] "RemoveContainer" containerID="51676a73d4dd485f021f44784500662f8c1074b6baefd3724687ea1d6b73b65e"
Oct 31 02:10:33.798773 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:10:33.798753    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51676a73d4dd485f021f44784500662f8c1074b6baefd3724687ea1d6b73b65e\": container with ID starting with 51676a73d4dd485f021f44784500662f8c1074b6baefd3724687ea1d6b73b65e not found: ID does not exist" containerID="51676a73d4dd485f021f44784500662f8c1074b6baefd3724687ea1d6b73b65e"
Oct 31 02:10:33.798856 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.798777    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51676a73d4dd485f021f44784500662f8c1074b6baefd3724687ea1d6b73b65e"} err="failed to get container status \"51676a73d4dd485f021f44784500662f8c1074b6baefd3724687ea1d6b73b65e\": rpc error: code = NotFound desc = could not find container \"51676a73d4dd485f021f44784500662f8c1074b6baefd3724687ea1d6b73b65e\": container with ID starting with 51676a73d4dd485f021f44784500662f8c1074b6baefd3724687ea1d6b73b65e not found: ID does not exist"
Oct 31 02:10:33.798856 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.798793    4476 scope.go:122] "RemoveContainer" containerID="ae972356d26187494eb58316cd76e3d9dafd9e1ac9fded1e3bc3940e89788eca"
Oct 31 02:10:33.806587 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.806571    4476 scope.go:122] "RemoveContainer" containerID="ae972356d26187494eb58316cd76e3d9dafd9e1ac9fded1e3bc3940e89788eca"
Oct 31 02:10:33.806828 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:10:33.806812    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae972356d26187494eb58316cd76e3d9dafd9e1ac9fded1e3bc3940e89788eca\": container with ID starting with ae972356d26187494eb58316cd76e3d9dafd9e1ac9fded1e3bc3940e89788eca not found: ID does not exist" containerID="ae972356d26187494eb58316cd76e3d9dafd9e1ac9fded1e3bc3940e89788eca"
Oct 31 02:10:33.806889 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.806832    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae972356d26187494eb58316cd76e3d9dafd9e1ac9fded1e3bc3940e89788eca"} err="failed to get container status \"ae972356d26187494eb58316cd76e3d9dafd9e1ac9fded1e3bc3940e89788eca\": rpc error: code = NotFound desc = could not find container \"ae972356d26187494eb58316cd76e3d9dafd9e1ac9fded1e3bc3940e89788eca\": container with ID starting with ae972356d26187494eb58316cd76e3d9dafd9e1ac9fded1e3bc3940e89788eca not found: ID does not exist"
Oct 31 02:10:33.944890 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.944863    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"]
Oct 31 02:10:33.947023 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:33.947004    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/simulator-w1-97b317-kserve-74d844d897-vsrgk"]
Oct 31 02:10:34.055933 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:34.055867    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"]
Oct 31 02:10:34.059800 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:34.059770    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/distinct-2-w1-97b317-kserve-85c8bdbfd4-b7vh4"]
Oct 31 02:10:34.174817 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:34.174787    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"]
Oct 31 02:10:34.179468 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:34.179447    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-97b317/distinct-w1-97b317-kserve-5566f4dc75-str28"]
Oct 31 02:10:34.453738 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:34.453659    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27377642-d40d-4498-af5d-16ca6a58ef54" path="/var/lib/kubelet/pods/27377642-d40d-4498-af5d-16ca6a58ef54/volumes"
Oct 31 02:10:34.454077 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:34.454064    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762230e3-0692-4a5b-8e29-13bf8a3a4b3c" path="/var/lib/kubelet/pods/762230e3-0692-4a5b-8e29-13bf8a3a4b3c/volumes"
Oct 31 02:10:34.454462 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:34.454449    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" path="/var/lib/kubelet/pods/cbdc50dd-29d0-4610-a361-cfd11ad1f501/volumes"
Oct 31 02:10:36.726108 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:36.726075    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:36.904610 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:36.904578    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:37.172809 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:37.172779    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:41.897479 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:41.897399    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:44.256476 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.256438    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv" podUID="9dcbe11f-bf2c-4e64-91db-96868f1f38ea" containerName="main" containerID="cri-o://e7db36c19eebde364216ad894264f07e6286f0ff6506c2a35785b3d4fea73948" gracePeriod=45
Oct 31 02:10:44.260567 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.260534    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerName="main" containerID="cri-o://8e854702261c76f9b0cd85ef7662939f074a99d4a7f996f3205f42b9d617ab28" gracePeriod=45
Oct 31 02:10:44.540784 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.540759    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 02:10:44.544447 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.544427    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 02:10:44.571876 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.571848    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-dshm\") pod \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") "
Oct 31 02:10:44.572007 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.571894    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-home\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-home\") pod \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") "
Oct 31 02:10:44.572007 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.571930    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-home\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-home\") pod \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") "
Oct 31 02:10:44.572007 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.571971    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-model-cache\") pod \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") "
Oct 31 02:10:44.572007 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572000    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/912dfe65-0c1e-4445-9d00-cf197ad4a93b-tls-certs\" (UniqueName: \"kubernetes.io/secret/912dfe65-0c1e-4445-9d00-cf197ad4a93b-tls-certs\") pod \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") "
Oct 31 02:10:44.572256 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572036    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-dshm\") pod \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") "
Oct 31 02:10:44.572256 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572094    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-model-cache\") pod \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") "
Oct 31 02:10:44.572256 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572130    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-tmp-dir\") pod \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") "
Oct 31 02:10:44.572256 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572159    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-kube-api-access-cg97w\" (UniqueName: \"kubernetes.io/projected/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-kube-api-access-cg97w\") pod \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") "
Oct 31 02:10:44.572256 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572237    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912dfe65-0c1e-4445-9d00-cf197ad4a93b-kube-api-access-4sv6h\" (UniqueName: \"kubernetes.io/projected/912dfe65-0c1e-4445-9d00-cf197ad4a93b-kube-api-access-4sv6h\") pod \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") "
Oct 31 02:10:44.572516 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572301    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-tls-certs\") pod \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\" (UID: \"9dcbe11f-bf2c-4e64-91db-96868f1f38ea\") "
Oct 31 02:10:44.572516 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572401    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-tmp-dir\") pod \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\" (UID: \"912dfe65-0c1e-4445-9d00-cf197ad4a93b\") "
Oct 31 02:10:44.572516 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572231    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-home" pod "9dcbe11f-bf2c-4e64-91db-96868f1f38ea" (UID: "9dcbe11f-bf2c-4e64-91db-96868f1f38ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:44.572516 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572476    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-tmp-dir" pod "9dcbe11f-bf2c-4e64-91db-96868f1f38ea" (UID: "9dcbe11f-bf2c-4e64-91db-96868f1f38ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:44.572516 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572241    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-model-cache" pod "912dfe65-0c1e-4445-9d00-cf197ad4a93b" (UID: "912dfe65-0c1e-4445-9d00-cf197ad4a93b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:44.572516 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572290    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-home" pod "912dfe65-0c1e-4445-9d00-cf197ad4a93b" (UID: "912dfe65-0c1e-4445-9d00-cf197ad4a93b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:44.572516 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572463    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-model-cache" pod "9dcbe11f-bf2c-4e64-91db-96868f1f38ea" (UID: "9dcbe11f-bf2c-4e64-91db-96868f1f38ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:44.572886 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572703    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-tmp-dir" pod "912dfe65-0c1e-4445-9d00-cf197ad4a93b" (UID: "912dfe65-0c1e-4445-9d00-cf197ad4a93b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:44.573198 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.572982    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.573198 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.573070    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.573198 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.573084    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.573198 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.573097    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.573198 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.573109    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.573198 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.573124    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.574669 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.574638    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912dfe65-0c1e-4445-9d00-cf197ad4a93b-kube-api-access-4sv6h" pod "912dfe65-0c1e-4445-9d00-cf197ad4a93b" (UID: "912dfe65-0c1e-4445-9d00-cf197ad4a93b"). InnerVolumeSpecName "kube-api-access-4sv6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:44.574761 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.574662    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-kube-api-access-cg97w" pod "9dcbe11f-bf2c-4e64-91db-96868f1f38ea" (UID: "9dcbe11f-bf2c-4e64-91db-96868f1f38ea"). InnerVolumeSpecName "kube-api-access-cg97w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:44.574807 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.574769    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-dshm" pod "9dcbe11f-bf2c-4e64-91db-96868f1f38ea" (UID: "9dcbe11f-bf2c-4e64-91db-96868f1f38ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:44.574807 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.574774    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-dshm" pod "912dfe65-0c1e-4445-9d00-cf197ad4a93b" (UID: "912dfe65-0c1e-4445-9d00-cf197ad4a93b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:44.575242 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.575219    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-tls-certs" pod "9dcbe11f-bf2c-4e64-91db-96868f1f38ea" (UID: "9dcbe11f-bf2c-4e64-91db-96868f1f38ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:44.575242 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.575223    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/912dfe65-0c1e-4445-9d00-cf197ad4a93b-tls-certs" pod "912dfe65-0c1e-4445-9d00-cf197ad4a93b" (UID: "912dfe65-0c1e-4445-9d00-cf197ad4a93b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:44.673993 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.673964    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/912dfe65-0c1e-4445-9d00-cf197ad4a93b-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.673993 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.673993    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.674222 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.674008    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg97w\" (UniqueName: \"kubernetes.io/projected/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-kube-api-access-cg97w\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.674222 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.674021    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sv6h\" (UniqueName: \"kubernetes.io/projected/912dfe65-0c1e-4445-9d00-cf197ad4a93b-kube-api-access-4sv6h\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.674222 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.674034    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dcbe11f-bf2c-4e64-91db-96868f1f38ea-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.674222 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.674045    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/912dfe65-0c1e-4445-9d00-cf197ad4a93b-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:44.844874 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.844840    4476 generic.go:356] "Generic (PLEG): container finished" podID="9dcbe11f-bf2c-4e64-91db-96868f1f38ea" containerID="e7db36c19eebde364216ad894264f07e6286f0ff6506c2a35785b3d4fea73948" exitCode=0
Oct 31 02:10:44.845079 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.844896    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"
Oct 31 02:10:44.845079 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.844935    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv" event={"ID":"9dcbe11f-bf2c-4e64-91db-96868f1f38ea","Type":"ContainerDied","Data":"e7db36c19eebde364216ad894264f07e6286f0ff6506c2a35785b3d4fea73948"}
Oct 31 02:10:44.845079 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.844987    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv" event={"ID":"9dcbe11f-bf2c-4e64-91db-96868f1f38ea","Type":"ContainerDied","Data":"34518805170b3aeb2d23cf5857e08d6b3c6e7660ac5de972829c9c5d6170c679"}
Oct 31 02:10:44.845079 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.845013    4476 scope.go:122] "RemoveContainer" containerID="e7db36c19eebde364216ad894264f07e6286f0ff6506c2a35785b3d4fea73948"
Oct 31 02:10:44.846581 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.846560    4476 generic.go:356] "Generic (PLEG): container finished" podID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerID="8e854702261c76f9b0cd85ef7662939f074a99d4a7f996f3205f42b9d617ab28" exitCode=0
Oct 31 02:10:44.846707 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.846626    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" event={"ID":"912dfe65-0c1e-4445-9d00-cf197ad4a93b","Type":"ContainerDied","Data":"8e854702261c76f9b0cd85ef7662939f074a99d4a7f996f3205f42b9d617ab28"}
Oct 31 02:10:44.846707 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.846646    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467" event={"ID":"912dfe65-0c1e-4445-9d00-cf197ad4a93b","Type":"ContainerDied","Data":"316e97c9b606c2c2aca3b1584e753cd8c506eb04663a711dd6d96b8938ef403f"}
Oct 31 02:10:44.846707 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.846691    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"
Oct 31 02:10:44.859400 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.858551    4476 scope.go:122] "RemoveContainer" containerID="e7db36c19eebde364216ad894264f07e6286f0ff6506c2a35785b3d4fea73948"
Oct 31 02:10:44.859400 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:10:44.858909    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7db36c19eebde364216ad894264f07e6286f0ff6506c2a35785b3d4fea73948\": container with ID starting with e7db36c19eebde364216ad894264f07e6286f0ff6506c2a35785b3d4fea73948 not found: ID does not exist" containerID="e7db36c19eebde364216ad894264f07e6286f0ff6506c2a35785b3d4fea73948"
Oct 31 02:10:44.859400 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.858938    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7db36c19eebde364216ad894264f07e6286f0ff6506c2a35785b3d4fea73948"} err="failed to get container status \"e7db36c19eebde364216ad894264f07e6286f0ff6506c2a35785b3d4fea73948\": rpc error: code = NotFound desc = could not find container \"e7db36c19eebde364216ad894264f07e6286f0ff6506c2a35785b3d4fea73948\": container with ID starting with e7db36c19eebde364216ad894264f07e6286f0ff6506c2a35785b3d4fea73948 not found: ID does not exist"
Oct 31 02:10:44.859400 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.858962    4476 scope.go:122] "RemoveContainer" containerID="8e854702261c76f9b0cd85ef7662939f074a99d4a7f996f3205f42b9d617ab28"
Oct 31 02:10:44.869015 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.868997    4476 scope.go:122] "RemoveContainer" containerID="8e854702261c76f9b0cd85ef7662939f074a99d4a7f996f3205f42b9d617ab28"
Oct 31 02:10:44.869881 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:10:44.869858    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e854702261c76f9b0cd85ef7662939f074a99d4a7f996f3205f42b9d617ab28\": container with ID starting with 8e854702261c76f9b0cd85ef7662939f074a99d4a7f996f3205f42b9d617ab28 not found: ID does not exist" containerID="8e854702261c76f9b0cd85ef7662939f074a99d4a7f996f3205f42b9d617ab28"
Oct 31 02:10:44.869962 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:44.869891    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e854702261c76f9b0cd85ef7662939f074a99d4a7f996f3205f42b9d617ab28"} err="failed to get container status \"8e854702261c76f9b0cd85ef7662939f074a99d4a7f996f3205f42b9d617ab28\": rpc error: code = NotFound desc = could not find container \"8e854702261c76f9b0cd85ef7662939f074a99d4a7f996f3205f42b9d617ab28\": container with ID starting with 8e854702261c76f9b0cd85ef7662939f074a99d4a7f996f3205f42b9d617ab28 not found: ID does not exist"
Oct 31 02:10:45.010794 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:45.010765    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"]
Oct 31 02:10:45.014877 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:45.014853    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/distinct-2-w3-ec695b-kserve-5d6f755b6c-6qxmv"]
Oct 31 02:10:45.122858 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:45.122787    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"]
Oct 31 02:10:45.127225 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:45.127202    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ec695b/distinct-w3-ec695b-kserve-669f64756-fr467"]
Oct 31 02:10:46.453334 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:46.453303    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" path="/var/lib/kubelet/pods/912dfe65-0c1e-4445-9d00-cf197ad4a93b/volumes"
Oct 31 02:10:46.453728 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:46.453718    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dcbe11f-bf2c-4e64-91db-96868f1f38ea" path="/var/lib/kubelet/pods/9dcbe11f-bf2c-4e64-91db-96868f1f38ea/volumes"
Oct 31 02:10:46.993196 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:46.993141    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:52.697668 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:52.697640    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:53.296379 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:53.296348    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:53.486817 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:53.486784    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:54.285491 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:54.285464    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:54.586417 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:54.586302    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:54.798349 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:54.798311    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:55.002166 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:55.002093    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:55.247629 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:55.247597    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:55.451962 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:55.451922    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:56.292751 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:56.292685    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:56.696749 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:56.696673    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:57.645906 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:57.645458    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:59.180748 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:10:59.180712    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:10:59.504544 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:10:59.504469    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:02.445100 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:02.445071    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:03.402447 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:03.402418    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:06.425133 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:06.425100    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:06.602878 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:06.602851    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:07.194859 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:07.194831    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:07.421150 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:07.421118    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:11.099423 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.099385    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"]
Oct 31 02:11:11.099849 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.099620    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5" podUID="3734f793-a26d-4f0b-8f5a-954b10bb51c7" containerName="maas-api" containerID="cri-o://5948020fbe3df2043cf240ecf4e4b3eb8a581e26033a41e4c347ce61cc2ade19" gracePeriod=30
Oct 31 02:11:11.244816 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.244780    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:11.353511 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.353450    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 02:11:11.548124 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.548097    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3734f793-a26d-4f0b-8f5a-954b10bb51c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3734f793-a26d-4f0b-8f5a-954b10bb51c7-maas-api-tls\") pod \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") "
Oct 31 02:11:11.548124 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.548129    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3734f793-a26d-4f0b-8f5a-954b10bb51c7-kube-api-access-pskbn\" (UniqueName: \"kubernetes.io/projected/3734f793-a26d-4f0b-8f5a-954b10bb51c7-kube-api-access-pskbn\") pod \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") "
Oct 31 02:11:11.548380 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.548156    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3734f793-a26d-4f0b-8f5a-954b10bb51c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3734f793-a26d-4f0b-8f5a-954b10bb51c7-openshift-service-ca\") pod \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") "
Oct 31 02:11:11.548380 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.548277    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3734f793-a26d-4f0b-8f5a-954b10bb51c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3734f793-a26d-4f0b-8f5a-954b10bb51c7-metrics-certs\") pod \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\" (UID: \"3734f793-a26d-4f0b-8f5a-954b10bb51c7\") "
Oct 31 02:11:11.548617 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.548591    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3734f793-a26d-4f0b-8f5a-954b10bb51c7-openshift-service-ca" pod "3734f793-a26d-4f0b-8f5a-954b10bb51c7" (UID: "3734f793-a26d-4f0b-8f5a-954b10bb51c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:11.548759 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.548743    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3734f793-a26d-4f0b-8f5a-954b10bb51c7-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:11.550606 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.550574    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3734f793-a26d-4f0b-8f5a-954b10bb51c7-maas-api-tls" pod "3734f793-a26d-4f0b-8f5a-954b10bb51c7" (UID: "3734f793-a26d-4f0b-8f5a-954b10bb51c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:11.550714 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.550620    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3734f793-a26d-4f0b-8f5a-954b10bb51c7-metrics-certs" pod "3734f793-a26d-4f0b-8f5a-954b10bb51c7" (UID: "3734f793-a26d-4f0b-8f5a-954b10bb51c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:11.550714 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.550644    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3734f793-a26d-4f0b-8f5a-954b10bb51c7-kube-api-access-pskbn" pod "3734f793-a26d-4f0b-8f5a-954b10bb51c7" (UID: "3734f793-a26d-4f0b-8f5a-954b10bb51c7"). InnerVolumeSpecName "kube-api-access-pskbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:11.649372 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.649284    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3734f793-a26d-4f0b-8f5a-954b10bb51c7-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:11.649372 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.649315    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3734f793-a26d-4f0b-8f5a-954b10bb51c7-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:11.649372 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.649330    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pskbn\" (UniqueName: \"kubernetes.io/projected/3734f793-a26d-4f0b-8f5a-954b10bb51c7-kube-api-access-pskbn\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:11.979997 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.979914    4476 generic.go:356] "Generic (PLEG): container finished" podID="3734f793-a26d-4f0b-8f5a-954b10bb51c7" containerID="5948020fbe3df2043cf240ecf4e4b3eb8a581e26033a41e4c347ce61cc2ade19" exitCode=0
Oct 31 02:11:11.979997 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.979962    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"
Oct 31 02:11:11.980215 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.980000    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5" event={"ID":"3734f793-a26d-4f0b-8f5a-954b10bb51c7","Type":"ContainerDied","Data":"5948020fbe3df2043cf240ecf4e4b3eb8a581e26033a41e4c347ce61cc2ade19"}
Oct 31 02:11:11.980215 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.980057    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5" event={"ID":"3734f793-a26d-4f0b-8f5a-954b10bb51c7","Type":"ContainerDied","Data":"d605686330f1b4f8007dc91235a3a8226c6af26ba2088d017f818e47ad4309e7"}
Oct 31 02:11:11.980215 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.980088    4476 scope.go:122] "RemoveContainer" containerID="5948020fbe3df2043cf240ecf4e4b3eb8a581e26033a41e4c347ce61cc2ade19"
Oct 31 02:11:11.990264 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.990246    4476 scope.go:122] "RemoveContainer" containerID="5948020fbe3df2043cf240ecf4e4b3eb8a581e26033a41e4c347ce61cc2ade19"
Oct 31 02:11:11.990498 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:11:11.990480    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5948020fbe3df2043cf240ecf4e4b3eb8a581e26033a41e4c347ce61cc2ade19\": container with ID starting with 5948020fbe3df2043cf240ecf4e4b3eb8a581e26033a41e4c347ce61cc2ade19 not found: ID does not exist" containerID="5948020fbe3df2043cf240ecf4e4b3eb8a581e26033a41e4c347ce61cc2ade19"
Oct 31 02:11:11.990551 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:11.990505    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5948020fbe3df2043cf240ecf4e4b3eb8a581e26033a41e4c347ce61cc2ade19"} err="failed to get container status \"5948020fbe3df2043cf240ecf4e4b3eb8a581e26033a41e4c347ce61cc2ade19\": rpc error: code = NotFound desc = could not find container \"5948020fbe3df2043cf240ecf4e4b3eb8a581e26033a41e4c347ce61cc2ade19\": container with ID starting with 5948020fbe3df2043cf240ecf4e4b3eb8a581e26033a41e4c347ce61cc2ade19 not found: ID does not exist"
Oct 31 02:11:12.142158 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:12.142124    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"]
Oct 31 02:11:12.191667 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:12.191639    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-97b317-6cdb577987-8tmc5"]
Oct 31 02:11:12.453494 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:12.453464    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3734f793-a26d-4f0b-8f5a-954b10bb51c7" path="/var/lib/kubelet/pods/3734f793-a26d-4f0b-8f5a-954b10bb51c7/volumes"
Oct 31 02:11:12.750963 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:12.750885    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:13.006222 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:13.006117    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:13.215268 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:13.215212    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:13.459329 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:13.459296    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:13.844907 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:13.844878    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:14.039746 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:14.039716    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:18.084721 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:18.084683    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:20.631057 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:20.631021    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:20.819569 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:20.819542    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:20.910939 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:20.910851    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"]
Oct 31 02:11:20.911108 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:20.911061    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc" podUID="69028e29-6686-4cd4-bf5a-94446d2c70d2" containerName="istio-proxy" containerID="cri-o://8da72dbc6bd118df10edc74b9e44e2d0f14b3bc02c77627c072c23e0ad6a1329" gracePeriod=30
Oct 31 02:11:21.055582 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:21.055546    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:23.021138 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:23.021106    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:23.219002 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:23.218969    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:24.097211 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:24.097163    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:24.290892 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:24.290859    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:25.645253 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:25.645189    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:26.065447 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.065404    4476 generic.go:356] "Generic (PLEG): container finished" podID="69028e29-6686-4cd4-bf5a-94446d2c70d2" containerID="8da72dbc6bd118df10edc74b9e44e2d0f14b3bc02c77627c072c23e0ad6a1329" exitCode=0
Oct 31 02:11:26.065657 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.065477    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc" event={"ID":"69028e29-6686-4cd4-bf5a-94446d2c70d2","Type":"ContainerDied","Data":"8da72dbc6bd118df10edc74b9e44e2d0f14b3bc02c77627c072c23e0ad6a1329"}
Oct 31 02:11:26.142186 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.142150    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:26.173354 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.173322    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 02:11:26.198913 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.198852    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-data\") pod \"69028e29-6686-4cd4-bf5a-94446d2c70d2\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") "
Oct 31 02:11:26.198913 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.198901    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69028e29-6686-4cd4-bf5a-94446d2c70d2-kube-api-access-bxhvb\" (UniqueName: \"kubernetes.io/projected/69028e29-6686-4cd4-bf5a-94446d2c70d2-kube-api-access-bxhvb\") pod \"69028e29-6686-4cd4-bf5a-94446d2c70d2\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") "
Oct 31 02:11:26.199115 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.198950    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69028e29-6686-4cd4-bf5a-94446d2c70d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69028e29-6686-4cd4-bf5a-94446d2c70d2-istiod-ca-cert\") pod \"69028e29-6686-4cd4-bf5a-94446d2c70d2\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") "
Oct 31 02:11:26.199115 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.198984    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-workload-socket\") pod \"69028e29-6686-4cd4-bf5a-94446d2c70d2\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") "
Oct 31 02:11:26.199115 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.199058    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-podinfo\") pod \"69028e29-6686-4cd4-bf5a-94446d2c70d2\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") "
Oct 31 02:11:26.199115 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.199100    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-workload-certs\") pod \"69028e29-6686-4cd4-bf5a-94446d2c70d2\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") "
Oct 31 02:11:26.199362 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.199128    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-envoy\") pod \"69028e29-6686-4cd4-bf5a-94446d2c70d2\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") "
Oct 31 02:11:26.199362 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.199150    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-data" pod "69028e29-6686-4cd4-bf5a-94446d2c70d2" (UID: "69028e29-6686-4cd4-bf5a-94446d2c70d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:26.199362 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.199338    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-workload-socket" pod "69028e29-6686-4cd4-bf5a-94446d2c70d2" (UID: "69028e29-6686-4cd4-bf5a-94446d2c70d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:26.199362 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.199349    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-credential-socket\") pod \"69028e29-6686-4cd4-bf5a-94446d2c70d2\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") "
Oct 31 02:11:26.199588 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.199386    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69028e29-6686-4cd4-bf5a-94446d2c70d2-istiod-ca-cert" pod "69028e29-6686-4cd4-bf5a-94446d2c70d2" (UID: "69028e29-6686-4cd4-bf5a-94446d2c70d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:26.199588 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.199402    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-token\" (UniqueName: \"kubernetes.io/projected/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-token\") pod \"69028e29-6686-4cd4-bf5a-94446d2c70d2\" (UID: \"69028e29-6686-4cd4-bf5a-94446d2c70d2\") "
Oct 31 02:11:26.199588 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.199463    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-workload-certs" pod "69028e29-6686-4cd4-bf5a-94446d2c70d2" (UID: "69028e29-6686-4cd4-bf5a-94446d2c70d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:26.199753 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.199615    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-credential-socket" pod "69028e29-6686-4cd4-bf5a-94446d2c70d2" (UID: "69028e29-6686-4cd4-bf5a-94446d2c70d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:26.200009 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.199987    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69028e29-6686-4cd4-bf5a-94446d2c70d2-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:26.200074 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.200017    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:26.200074 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.200034    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:26.200074 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.200048    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:26.200074 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.200061    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:26.201446 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.201419    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69028e29-6686-4cd4-bf5a-94446d2c70d2-kube-api-access-bxhvb" pod "69028e29-6686-4cd4-bf5a-94446d2c70d2" (UID: "69028e29-6686-4cd4-bf5a-94446d2c70d2"). InnerVolumeSpecName "kube-api-access-bxhvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:26.201446 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.201424    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-envoy" pod "69028e29-6686-4cd4-bf5a-94446d2c70d2" (UID: "69028e29-6686-4cd4-bf5a-94446d2c70d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:26.201654 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.201464    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-podinfo" pod "69028e29-6686-4cd4-bf5a-94446d2c70d2" (UID: "69028e29-6686-4cd4-bf5a-94446d2c70d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:11:26.201765 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.201749    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-token" pod "69028e29-6686-4cd4-bf5a-94446d2c70d2" (UID: "69028e29-6686-4cd4-bf5a-94446d2c70d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:26.301421 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.301381    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:26.301421 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.301421    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:26.301639 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.301435    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69028e29-6686-4cd4-bf5a-94446d2c70d2-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:26.301639 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.301450    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxhvb\" (UniqueName: \"kubernetes.io/projected/69028e29-6686-4cd4-bf5a-94446d2c70d2-kube-api-access-bxhvb\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:26.341785 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.341748    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:26.518138 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:26.518106    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:27.071816 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:27.071791    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"
Oct 31 02:11:27.072324 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:27.071831    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc" event={"ID":"69028e29-6686-4cd4-bf5a-94446d2c70d2","Type":"ContainerDied","Data":"8c3daed49e430ce658d2ed370f5313429087ee93094a71102a91ef7ca60cbb0f"}
Oct 31 02:11:27.072324 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:27.071900    4476 scope.go:122] "RemoveContainer" containerID="8da72dbc6bd118df10edc74b9e44e2d0f14b3bc02c77627c072c23e0ad6a1329"
Oct 31 02:11:27.291535 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:27.291499    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"]
Oct 31 02:11:27.296487 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:27.296463    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:27.341148 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:27.341072    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-m4rsc"]
Oct 31 02:11:27.652852 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:27.652772    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:27.853955 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:27.853920    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:28.363049 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.363008    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"]
Oct 31 02:11:28.363581 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.363285    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9" podUID="5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" containerName="maas-api" containerID="cri-o://64d4df4ff64bbd979fa9e986ce20dd75ecd98372346c4bad20666484b34cb218" gracePeriod=30
Oct 31 02:11:28.453618 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.453580    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69028e29-6686-4cd4-bf5a-94446d2c70d2" path="/var/lib/kubelet/pods/69028e29-6686-4cd4-bf5a-94446d2c70d2/volumes"
Oct 31 02:11:28.494958 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.494924    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9" podUID="5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": dial tcp 10.133.0.59:8443: connect: connection refused"
Oct 31 02:11:28.513235 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.513201    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:28.632161 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.632138    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 02:11:28.704680 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.704653    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:28.726772 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.726749    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-openshift-service-ca\") pod \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") "
Oct 31 02:11:28.726900 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.726810    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-maas-api-tls\") pod \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") "
Oct 31 02:11:28.726900 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.726868    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-kube-api-access-prkcf\" (UniqueName: \"kubernetes.io/projected/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-kube-api-access-prkcf\") pod \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") "
Oct 31 02:11:28.726974 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.726900    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-metrics-certs\") pod \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\" (UID: \"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5\") "
Oct 31 02:11:28.727102 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.727081    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-openshift-service-ca" pod "5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" (UID: "5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:28.727218 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.727200    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:28.729156 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.729134    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-metrics-certs" pod "5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" (UID: "5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:28.729286 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.729193    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-maas-api-tls" pod "5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" (UID: "5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:28.729331 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.729293    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-kube-api-access-prkcf" pod "5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" (UID: "5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5"). InnerVolumeSpecName "kube-api-access-prkcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:28.828478 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.828446    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:28.828478 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.828473    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prkcf\" (UniqueName: \"kubernetes.io/projected/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-kube-api-access-prkcf\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:28.828478 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.828484    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:28.904382 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:28.904316    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:29.085382 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.085345    4476 generic.go:356] "Generic (PLEG): container finished" podID="5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" containerID="64d4df4ff64bbd979fa9e986ce20dd75ecd98372346c4bad20666484b34cb218" exitCode=0
Oct 31 02:11:29.085382 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.085369    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9" event={"ID":"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5","Type":"ContainerDied","Data":"64d4df4ff64bbd979fa9e986ce20dd75ecd98372346c4bad20666484b34cb218"}
Oct 31 02:11:29.085646 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.085420    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9" event={"ID":"5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5","Type":"ContainerDied","Data":"544b534580dd41bfb0f3d6a87364d2c738f97451858528a81501211dfd75e4d2"}
Oct 31 02:11:29.085646 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.085442    4476 scope.go:122] "RemoveContainer" containerID="64d4df4ff64bbd979fa9e986ce20dd75ecd98372346c4bad20666484b34cb218"
Oct 31 02:11:29.085646 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.085381    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"
Oct 31 02:11:29.094934 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.094909    4476 scope.go:122] "RemoveContainer" containerID="2d8b5e6dcd8579ec079396d1d16cd4fd2ea6dc4614b95c5c0a8784cac0713246"
Oct 31 02:11:29.103755 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.103737    4476 scope.go:122] "RemoveContainer" containerID="64d4df4ff64bbd979fa9e986ce20dd75ecd98372346c4bad20666484b34cb218"
Oct 31 02:11:29.104011 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:11:29.103992    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d4df4ff64bbd979fa9e986ce20dd75ecd98372346c4bad20666484b34cb218\": container with ID starting with 64d4df4ff64bbd979fa9e986ce20dd75ecd98372346c4bad20666484b34cb218 not found: ID does not exist" containerID="64d4df4ff64bbd979fa9e986ce20dd75ecd98372346c4bad20666484b34cb218"
Oct 31 02:11:29.104053 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.104020    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d4df4ff64bbd979fa9e986ce20dd75ecd98372346c4bad20666484b34cb218"} err="failed to get container status \"64d4df4ff64bbd979fa9e986ce20dd75ecd98372346c4bad20666484b34cb218\": rpc error: code = NotFound desc = could not find container \"64d4df4ff64bbd979fa9e986ce20dd75ecd98372346c4bad20666484b34cb218\": container with ID starting with 64d4df4ff64bbd979fa9e986ce20dd75ecd98372346c4bad20666484b34cb218 not found: ID does not exist"
Oct 31 02:11:29.104053 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.104038    4476 scope.go:122] "RemoveContainer" containerID="2d8b5e6dcd8579ec079396d1d16cd4fd2ea6dc4614b95c5c0a8784cac0713246"
Oct 31 02:11:29.104286 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:11:29.104269    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d8b5e6dcd8579ec079396d1d16cd4fd2ea6dc4614b95c5c0a8784cac0713246\": container with ID starting with 2d8b5e6dcd8579ec079396d1d16cd4fd2ea6dc4614b95c5c0a8784cac0713246 not found: ID does not exist" containerID="2d8b5e6dcd8579ec079396d1d16cd4fd2ea6dc4614b95c5c0a8784cac0713246"
Oct 31 02:11:29.104338 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.104294    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d8b5e6dcd8579ec079396d1d16cd4fd2ea6dc4614b95c5c0a8784cac0713246"} err="failed to get container status \"2d8b5e6dcd8579ec079396d1d16cd4fd2ea6dc4614b95c5c0a8784cac0713246\": rpc error: code = NotFound desc = could not find container \"2d8b5e6dcd8579ec079396d1d16cd4fd2ea6dc4614b95c5c0a8784cac0713246\": container with ID starting with 2d8b5e6dcd8579ec079396d1d16cd4fd2ea6dc4614b95c5c0a8784cac0713246 not found: ID does not exist"
Oct 31 02:11:29.182459 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:11:29.182369    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:11:29.207160 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.207132    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:29.246934 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.246900    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"]
Oct 31 02:11:29.252501 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.252476    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e621db-7768d56ff5-qlxq9"]
Oct 31 02:11:29.554054 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.554019    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:29.860397 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:29.860312    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:30.043376 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:30.043340    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:30.224362 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:30.224284    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:30.441913 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:30.441885    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:30.452624 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:30.452598    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" path="/var/lib/kubelet/pods/5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5/volumes"
Oct 31 02:11:30.612908 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:30.612880    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:31.040713 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:31.040687    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 02:11:31.051164 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:31.051141    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 02:11:31.082463 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:31.082434    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:34.906322 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:34.906287    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:35.730301 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:35.730269    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:36.316577 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:36.316544    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:36.514557 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:36.514522    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:36.702895 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:36.702806    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:37.528800 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:37.528755    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:37.704879 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:37.704823    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:37.840930 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:37.840854    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"]
Oct 31 02:11:37.841120 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:37.841098    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5" podUID="c76cc5c8-d8e3-44d4-b419-9a38439489e2" containerName="istio-proxy" containerID="cri-o://dc856bc84de5955739c7a4380ead588c2a668b05e3de384469724bd7e5954dea" gracePeriod=30
Oct 31 02:11:37.901595 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:37.901557    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:40.028253 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:40.028225    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:40.200523 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:40.200491    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:40.585322 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:40.585268    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:43.094990 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.094964    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 02:11:43.159006 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.158920    4476 generic.go:356] "Generic (PLEG): container finished" podID="c76cc5c8-d8e3-44d4-b419-9a38439489e2" containerID="dc856bc84de5955739c7a4380ead588c2a668b05e3de384469724bd7e5954dea" exitCode=0
Oct 31 02:11:43.159264 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.158994    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"
Oct 31 02:11:43.159408 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.159012    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5" event={"ID":"c76cc5c8-d8e3-44d4-b419-9a38439489e2","Type":"ContainerDied","Data":"dc856bc84de5955739c7a4380ead588c2a668b05e3de384469724bd7e5954dea"}
Oct 31 02:11:43.159408 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.159378    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5" event={"ID":"c76cc5c8-d8e3-44d4-b419-9a38439489e2","Type":"ContainerDied","Data":"0c15bc6627c98c5c3188b10468f403c36a61e1b2d93ad7106eb9122391a910d9"}
Oct 31 02:11:43.159507 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.159408    4476 scope.go:122] "RemoveContainer" containerID="dc856bc84de5955739c7a4380ead588c2a668b05e3de384469724bd7e5954dea"
Oct 31 02:11:43.169960 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.169086    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-credential-socket\") pod \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") "
Oct 31 02:11:43.169960 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.169133    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-podinfo\") pod \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") "
Oct 31 02:11:43.169960 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.169161    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-workload-certs\") pod \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") "
Oct 31 02:11:43.169960 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.169210    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istiod-ca-cert\") pod \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") "
Oct 31 02:11:43.169960 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.169257    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-workload-socket\") pod \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") "
Oct 31 02:11:43.169960 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.169288    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-token\" (UniqueName: \"kubernetes.io/projected/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-token\") pod \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") "
Oct 31 02:11:43.169960 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.169351    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76cc5c8-d8e3-44d4-b419-9a38439489e2-kube-api-access-zmbgr\" (UniqueName: \"kubernetes.io/projected/c76cc5c8-d8e3-44d4-b419-9a38439489e2-kube-api-access-zmbgr\") pod \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") "
Oct 31 02:11:43.169960 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.169387    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-envoy\") pod \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") "
Oct 31 02:11:43.169960 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.169412    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-data\") pod \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\" (UID: \"c76cc5c8-d8e3-44d4-b419-9a38439489e2\") "
Oct 31 02:11:43.170581 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.170359    4476 scope.go:122] "RemoveContainer" containerID="dc856bc84de5955739c7a4380ead588c2a668b05e3de384469724bd7e5954dea"
Oct 31 02:11:43.170775 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.170748    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istiod-ca-cert" pod "c76cc5c8-d8e3-44d4-b419-9a38439489e2" (UID: "c76cc5c8-d8e3-44d4-b419-9a38439489e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:43.170775 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.170775    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-data" pod "c76cc5c8-d8e3-44d4-b419-9a38439489e2" (UID: "c76cc5c8-d8e3-44d4-b419-9a38439489e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:43.171088 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:11:43.171006    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc856bc84de5955739c7a4380ead588c2a668b05e3de384469724bd7e5954dea\": container with ID starting with dc856bc84de5955739c7a4380ead588c2a668b05e3de384469724bd7e5954dea not found: ID does not exist" containerID="dc856bc84de5955739c7a4380ead588c2a668b05e3de384469724bd7e5954dea"
Oct 31 02:11:43.171088 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.171049    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc856bc84de5955739c7a4380ead588c2a668b05e3de384469724bd7e5954dea"} err="failed to get container status \"dc856bc84de5955739c7a4380ead588c2a668b05e3de384469724bd7e5954dea\": rpc error: code = NotFound desc = could not find container \"dc856bc84de5955739c7a4380ead588c2a668b05e3de384469724bd7e5954dea\": container with ID starting with dc856bc84de5955739c7a4380ead588c2a668b05e3de384469724bd7e5954dea not found: ID does not exist"
Oct 31 02:11:43.171088 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.171015    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-credential-socket" pod "c76cc5c8-d8e3-44d4-b419-9a38439489e2" (UID: "c76cc5c8-d8e3-44d4-b419-9a38439489e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:43.171338 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.171123    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-workload-socket" pod "c76cc5c8-d8e3-44d4-b419-9a38439489e2" (UID: "c76cc5c8-d8e3-44d4-b419-9a38439489e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:43.171338 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.171150    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-workload-certs" pod "c76cc5c8-d8e3-44d4-b419-9a38439489e2" (UID: "c76cc5c8-d8e3-44d4-b419-9a38439489e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:43.172946 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.172918    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-token" pod "c76cc5c8-d8e3-44d4-b419-9a38439489e2" (UID: "c76cc5c8-d8e3-44d4-b419-9a38439489e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:43.173071 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.173055    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-envoy" pod "c76cc5c8-d8e3-44d4-b419-9a38439489e2" (UID: "c76cc5c8-d8e3-44d4-b419-9a38439489e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:43.173164 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.173143    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76cc5c8-d8e3-44d4-b419-9a38439489e2-kube-api-access-zmbgr" pod "c76cc5c8-d8e3-44d4-b419-9a38439489e2" (UID: "c76cc5c8-d8e3-44d4-b419-9a38439489e2"). InnerVolumeSpecName "kube-api-access-zmbgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:43.173556 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.173524    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-podinfo" pod "c76cc5c8-d8e3-44d4-b419-9a38439489e2" (UID: "c76cc5c8-d8e3-44d4-b419-9a38439489e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:11:43.270822 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.270787    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmbgr\" (UniqueName: \"kubernetes.io/projected/c76cc5c8-d8e3-44d4-b419-9a38439489e2-kube-api-access-zmbgr\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:43.270822 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.270815    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:43.270822 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.270825    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:43.271054 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.270834    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:43.271054 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.270843    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:43.271054 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.270853    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:43.271054 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.270862    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:43.271054 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.270871    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76cc5c8-d8e3-44d4-b419-9a38439489e2-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:43.271054 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.270879    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76cc5c8-d8e3-44d4-b419-9a38439489e2-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:43.579573 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.579537    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"]
Oct 31 02:11:43.583581 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:43.583558    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-lz6v5"]
Oct 31 02:11:44.453687 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:44.453652    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76cc5c8-d8e3-44d4-b419-9a38439489e2" path="/var/lib/kubelet/pods/c76cc5c8-d8e3-44d4-b419-9a38439489e2/volumes"
Oct 31 02:11:45.290712 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:45.290674    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:49.972442 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:49.972407    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:54.952734 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:54.952695    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:55.151396 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:55.151359    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:58.896104 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:11:58.896066    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:59.180856 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:11:59.180777    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:12:00.381823 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:00.381789    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:00.590138 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:00.590110    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:00.759806 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:00.759729    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:05.689059 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:05.689024    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:10.392540 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:10.392460    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:11.329318 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:11.329284    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:15.067114 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:15.067086    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:17.419140 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:17.419094    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:17.603442 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:17.603406    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:17.780226 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:17.780166    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:20.012606 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:20.012567    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:24.806291 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:24.806259    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:25.671958 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:25.671927    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:28.410962 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:28.410931    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:29.180898 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:12:29.180860    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:12:29.585762 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:29.585696    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:30.044059 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:30.044030    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:30.227383 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:30.227349    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:30.442565 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:30.442492    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:30.623592 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:30.623559    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:33.493989 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:33.493959    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:34.178914 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:34.178859    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:38.885081 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:38.885046    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:43.695441 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:43.695406    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:48.377619 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:48.377451    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:52.547502 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:52.547469    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:54.054345 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:54.054308    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:54.287875 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:54.287840    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:54.474210 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:54.474091    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:54.983334 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:54.983299    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:55.216711 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:55.216673    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:59.181692 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:12:59.181660    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:12:59.292367 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:12:59.292333    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:01.211763 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:01.211721    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:01.385927 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:01.385893    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:01.582147 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:01.582119    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:04.205313 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:04.205281    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:09.119588 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:09.119551    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:13.062195 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.062140    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"]
Oct 31 02:13:13.062640 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.062402    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl" podUID="eb736102-7fdb-47d2-93f9-fa3fa5d67ed5" containerName="maas-api" containerID="cri-o://237ef368ef249fa06430b23f6dcad3c042405ce07bb7f3d99056d22a74b54ac0" gracePeriod=30
Oct 31 02:13:13.204249 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.204218    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:13.308686 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.308661    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 02:13:13.341813 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.341745    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-openshift-service-ca\") pod \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") "
Oct 31 02:13:13.341813 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.341792    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-kube-api-access-ccd4s\" (UniqueName: \"kubernetes.io/projected/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-kube-api-access-ccd4s\") pod \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") "
Oct 31 02:13:13.342014 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.341836    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-metrics-certs\") pod \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") "
Oct 31 02:13:13.342014 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.341886    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-maas-api-tls\") pod \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\" (UID: \"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5\") "
Oct 31 02:13:13.342194 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.342151    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-openshift-service-ca" pod "eb736102-7fdb-47d2-93f9-fa3fa5d67ed5" (UID: "eb736102-7fdb-47d2-93f9-fa3fa5d67ed5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:13.342290 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.342271    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:13:13.344109 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.344079    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-maas-api-tls" pod "eb736102-7fdb-47d2-93f9-fa3fa5d67ed5" (UID: "eb736102-7fdb-47d2-93f9-fa3fa5d67ed5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:13:13.344237 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.344122    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-metrics-certs" pod "eb736102-7fdb-47d2-93f9-fa3fa5d67ed5" (UID: "eb736102-7fdb-47d2-93f9-fa3fa5d67ed5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:13:13.344237 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.344155    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-kube-api-access-ccd4s" pod "eb736102-7fdb-47d2-93f9-fa3fa5d67ed5" (UID: "eb736102-7fdb-47d2-93f9-fa3fa5d67ed5"). InnerVolumeSpecName "kube-api-access-ccd4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:13.443530 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.443491    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccd4s\" (UniqueName: \"kubernetes.io/projected/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-kube-api-access-ccd4s\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:13:13.443530 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.443518    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:13:13.443530 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.443528    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:13:13.558208 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.558155    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"]
Oct 31 02:13:13.589412 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.589377    4476 generic.go:356] "Generic (PLEG): container finished" podID="eb736102-7fdb-47d2-93f9-fa3fa5d67ed5" containerID="237ef368ef249fa06430b23f6dcad3c042405ce07bb7f3d99056d22a74b54ac0" exitCode=0
Oct 31 02:13:13.589583 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.589427    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"
Oct 31 02:13:13.589583 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.589455    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl" event={"ID":"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5","Type":"ContainerDied","Data":"237ef368ef249fa06430b23f6dcad3c042405ce07bb7f3d99056d22a74b54ac0"}
Oct 31 02:13:13.589583 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.589508    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl" event={"ID":"eb736102-7fdb-47d2-93f9-fa3fa5d67ed5","Type":"ContainerDied","Data":"fa02d18df262a9d6269835de07a9c54db1792a9617e62367a868c117d8c465f1"}
Oct 31 02:13:13.589583 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.589539    4476 scope.go:122] "RemoveContainer" containerID="237ef368ef249fa06430b23f6dcad3c042405ce07bb7f3d99056d22a74b54ac0"
Oct 31 02:13:13.606344 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.606311    4476 scope.go:122] "RemoveContainer" containerID="237ef368ef249fa06430b23f6dcad3c042405ce07bb7f3d99056d22a74b54ac0"
Oct 31 02:13:13.606707 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:13:13.606674    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237ef368ef249fa06430b23f6dcad3c042405ce07bb7f3d99056d22a74b54ac0\": container with ID starting with 237ef368ef249fa06430b23f6dcad3c042405ce07bb7f3d99056d22a74b54ac0 not found: ID does not exist" containerID="237ef368ef249fa06430b23f6dcad3c042405ce07bb7f3d99056d22a74b54ac0"
Oct 31 02:13:13.606817 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.606713    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237ef368ef249fa06430b23f6dcad3c042405ce07bb7f3d99056d22a74b54ac0"} err="failed to get container status \"237ef368ef249fa06430b23f6dcad3c042405ce07bb7f3d99056d22a74b54ac0\": rpc error: code = NotFound desc = could not find container \"237ef368ef249fa06430b23f6dcad3c042405ce07bb7f3d99056d22a74b54ac0\": container with ID starting with 237ef368ef249fa06430b23f6dcad3c042405ce07bb7f3d99056d22a74b54ac0 not found: ID does not exist"
Oct 31 02:13:13.693721 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.693695    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:13.780780 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.780749    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"]
Oct 31 02:13:13.782399 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.782376    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-b6f24b-7ff4bdc55d-hfjvl"]
Oct 31 02:13:13.872138 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:13.872063    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:14.452869 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:14.452835    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb736102-7fdb-47d2-93f9-fa3fa5d67ed5" path="/var/lib/kubelet/pods/eb736102-7fdb-47d2-93f9-fa3fa5d67ed5/volumes"
Oct 31 02:13:14.983094 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:14.983056    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:17.817723 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:17.817685    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:18.082797 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:18.082716    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:18.559516 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:18.559473    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz" podUID="c382414a-6f3f-41fc-b773-b315da186244" containerName="payload-pre-processing" containerID="cri-o://04df0c872c39388d4d7df2ae9d83a592f483a50319eb9ba6910b27fea4143a42" gracePeriod=25
Oct 31 02:13:18.734093 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:18.734055    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:19.825855 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:19.825826    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:24.529018 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:24.528981    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:24.722781 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:24.722744    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:24.901726 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:24.901577    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:25.626221 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:25.626164    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz" podUID="c382414a-6f3f-41fc-b773-b315da186244" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:13:29.180753 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:13:29.180702    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:13:29.672655 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:29.672619    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:33.826939 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:33.826906    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:34.438072 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:34.438036    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:34.897658 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:34.897630    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:35.625964 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:35.625920    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz" podUID="c382414a-6f3f-41fc-b773-b315da186244" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:13:37.636384 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:37.636352    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:37.812992 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:37.812957    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:38.706694 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:38.706656    4476 generic.go:356] "Generic (PLEG): container finished" podID="c382414a-6f3f-41fc-b773-b315da186244" containerID="04df0c872c39388d4d7df2ae9d83a592f483a50319eb9ba6910b27fea4143a42" exitCode=0
Oct 31 02:13:38.707166 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:38.706735    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz" event={"ID":"c382414a-6f3f-41fc-b773-b315da186244","Type":"ContainerDied","Data":"04df0c872c39388d4d7df2ae9d83a592f483a50319eb9ba6910b27fea4143a42"}
Oct 31 02:13:38.810681 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:38.810659    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"
Oct 31 02:13:38.881427 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:38.880967    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c382414a-6f3f-41fc-b773-b315da186244-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c382414a-6f3f-41fc-b773-b315da186244-plugins-config-volume\") pod \"c382414a-6f3f-41fc-b773-b315da186244\" (UID: \"c382414a-6f3f-41fc-b773-b315da186244\") "
Oct 31 02:13:38.881427 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:38.881090    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c382414a-6f3f-41fc-b773-b315da186244-kube-api-access-5rbsf\" (UniqueName: \"kubernetes.io/projected/c382414a-6f3f-41fc-b773-b315da186244-kube-api-access-5rbsf\") pod \"c382414a-6f3f-41fc-b773-b315da186244\" (UID: \"c382414a-6f3f-41fc-b773-b315da186244\") "
Oct 31 02:13:38.881732 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:38.881706    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c382414a-6f3f-41fc-b773-b315da186244-plugins-config-volume" pod "c382414a-6f3f-41fc-b773-b315da186244" (UID: "c382414a-6f3f-41fc-b773-b315da186244"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:38.883646 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:38.883611    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c382414a-6f3f-41fc-b773-b315da186244-kube-api-access-5rbsf" pod "c382414a-6f3f-41fc-b773-b315da186244" (UID: "c382414a-6f3f-41fc-b773-b315da186244"). InnerVolumeSpecName "kube-api-access-5rbsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:38.982702 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:38.982667    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c382414a-6f3f-41fc-b773-b315da186244-plugins-config-volume\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:13:38.982702 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:38.982695    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rbsf\" (UniqueName: \"kubernetes.io/projected/c382414a-6f3f-41fc-b773-b315da186244-kube-api-access-5rbsf\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:13:39.071507 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:39.071461    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:39.712903 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:39.712873    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"
Oct 31 02:13:39.713415 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:39.712902    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz" event={"ID":"c382414a-6f3f-41fc-b773-b315da186244","Type":"ContainerDied","Data":"2a7442e900e2f9a7cc9a8b1194e1e24391b517457eda7db7d47103854a7f914d"}
Oct 31 02:13:39.713415 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:39.712965    4476 scope.go:122] "RemoveContainer" containerID="04df0c872c39388d4d7df2ae9d83a592f483a50319eb9ba6910b27fea4143a42"
Oct 31 02:13:39.874480 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:39.874444    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"]
Oct 31 02:13:39.923917 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:39.923885    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b6f24b-6c5b6bc95f-9grfz"]
Oct 31 02:13:40.453721 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:40.453692    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c382414a-6f3f-41fc-b773-b315da186244" path="/var/lib/kubelet/pods/c382414a-6f3f-41fc-b773-b315da186244/volumes"
Oct 31 02:13:43.172520 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:43.172486    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:44.674611 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:44.674577    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:44.890445 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:44.890416    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:45.064399 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:45.064373    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:45.592442 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:45.592414    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:49.875618 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:49.875582    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:51.417627 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:51.417596    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:51.596937 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:51.596905    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:51.795676 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:51.795643    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:52.246202 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:52.246102    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:54.889043 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:54.889013    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:57.156717 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:57.156684    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"]
Oct 31 02:13:57.206317 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:57.206285    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"]
Oct 31 02:13:58.331498 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:58.331454    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:59.180618 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:13:59.180585    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:13:59.783519 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:13:59.783489    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:04.643123 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:04.643083    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:04.818875 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:04.818836    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:12.176836 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.176792    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main" containerID="cri-o://de0a692e3e09cccb263520487ccb5d3ca80a23ad26d39185f4619252b0c6a85c" gracePeriod=45
Oct 31 02:14:12.225613 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.225546    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" containerID="cri-o://e757081dce247a93711fa19a9d3fa20f1b8e72c6ab70d3fd5d2338e824e95083" gracePeriod=45
Oct 31 02:14:12.472587 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.472561    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 02:14:12.491397 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.491377    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 02:14:12.492557 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.492539    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-model-cache\") pod \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") "
Oct 31 02:14:12.492610 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.492581    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-home\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-home\") pod \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") "
Oct 31 02:14:12.492610 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.492607    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-tmp-dir\") pod \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") "
Oct 31 02:14:12.492689 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.492634    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-tls-certs\") pod \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") "
Oct 31 02:14:12.492689 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.492657    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-dshm\") pod \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") "
Oct 31 02:14:12.492800 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.492756    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-kube-api-access-cdl8q\" (UniqueName: \"kubernetes.io/projected/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-kube-api-access-cdl8q\") pod \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\" (UID: \"4a67d783-3ecc-4425-8e50-8b1e72f4d57b\") "
Oct 31 02:14:12.492857 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.492834    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-model-cache" pod "4a67d783-3ecc-4425-8e50-8b1e72f4d57b" (UID: "4a67d783-3ecc-4425-8e50-8b1e72f4d57b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:12.492912 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.492852    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-home" pod "4a67d783-3ecc-4425-8e50-8b1e72f4d57b" (UID: "4a67d783-3ecc-4425-8e50-8b1e72f4d57b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:12.492912 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.492874    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-tmp-dir" pod "4a67d783-3ecc-4425-8e50-8b1e72f4d57b" (UID: "4a67d783-3ecc-4425-8e50-8b1e72f4d57b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:12.493061 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.493048    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:12.493102 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.493064    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:12.493102 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.493074    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:12.494998 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.494975    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-kube-api-access-cdl8q" pod "4a67d783-3ecc-4425-8e50-8b1e72f4d57b" (UID: "4a67d783-3ecc-4425-8e50-8b1e72f4d57b"). InnerVolumeSpecName "kube-api-access-cdl8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:12.495096 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.495000    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-tls-certs" pod "4a67d783-3ecc-4425-8e50-8b1e72f4d57b" (UID: "4a67d783-3ecc-4425-8e50-8b1e72f4d57b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:14:12.495096 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.495004    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-dshm" pod "4a67d783-3ecc-4425-8e50-8b1e72f4d57b" (UID: "4a67d783-3ecc-4425-8e50-8b1e72f4d57b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:12.594028 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.593998    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-tmp-dir\") pod \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") "
Oct 31 02:14:12.594196 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.594086    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-kube-api-access-xs2ck\" (UniqueName: \"kubernetes.io/projected/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-kube-api-access-xs2ck\") pod \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") "
Oct 31 02:14:12.594196 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.594126    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-dshm\") pod \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") "
Oct 31 02:14:12.594196 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.594164    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-home\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-home\") pod \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") "
Oct 31 02:14:12.594329 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.594236    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-tmp-dir" pod "3b2d11a3-2014-4d3c-b1e6-c98233a16c74" (UID: "3b2d11a3-2014-4d3c-b1e6-c98233a16c74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:12.594329 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.594268    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-tls-certs\") pod \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") "
Oct 31 02:14:12.594329 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.594307    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-model-cache\") pod \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\" (UID: \"3b2d11a3-2014-4d3c-b1e6-c98233a16c74\") "
Oct 31 02:14:12.594467 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.594454    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-home" pod "3b2d11a3-2014-4d3c-b1e6-c98233a16c74" (UID: "3b2d11a3-2014-4d3c-b1e6-c98233a16c74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:12.594612 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.594591    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-model-cache" pod "3b2d11a3-2014-4d3c-b1e6-c98233a16c74" (UID: "3b2d11a3-2014-4d3c-b1e6-c98233a16c74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:12.594727 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.594713    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:12.594792 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.594735    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-tmp-dir\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:12.594792 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.594750    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:12.594792 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.594764    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-home\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:12.594792 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.594779    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdl8q\" (UniqueName: \"kubernetes.io/projected/4a67d783-3ecc-4425-8e50-8b1e72f4d57b-kube-api-access-cdl8q\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:12.594958 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.594794    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-model-cache\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:12.596578 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.596555    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-tls-certs" pod "3b2d11a3-2014-4d3c-b1e6-c98233a16c74" (UID: "3b2d11a3-2014-4d3c-b1e6-c98233a16c74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:14:12.596844 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.596823    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-dshm" pod "3b2d11a3-2014-4d3c-b1e6-c98233a16c74" (UID: "3b2d11a3-2014-4d3c-b1e6-c98233a16c74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:12.596844 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.596831    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-kube-api-access-xs2ck" pod "3b2d11a3-2014-4d3c-b1e6-c98233a16c74" (UID: "3b2d11a3-2014-4d3c-b1e6-c98233a16c74"). InnerVolumeSpecName "kube-api-access-xs2ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:12.695931 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.695895    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs2ck\" (UniqueName: \"kubernetes.io/projected/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-kube-api-access-xs2ck\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:12.695931 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.695930    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-dshm\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:12.696129 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.695940    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b2d11a3-2014-4d3c-b1e6-c98233a16c74-tls-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:12.864601 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.864564    4476 generic.go:356] "Generic (PLEG): container finished" podID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerID="de0a692e3e09cccb263520487ccb5d3ca80a23ad26d39185f4619252b0c6a85c" exitCode=0
Oct 31 02:14:12.864783 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.864608    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"
Oct 31 02:14:12.864783 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.864653    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" event={"ID":"4a67d783-3ecc-4425-8e50-8b1e72f4d57b","Type":"ContainerDied","Data":"de0a692e3e09cccb263520487ccb5d3ca80a23ad26d39185f4619252b0c6a85c"}
Oct 31 02:14:12.864783 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.864697    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx" event={"ID":"4a67d783-3ecc-4425-8e50-8b1e72f4d57b","Type":"ContainerDied","Data":"adb2387948723e3927bd54d5a63363a92b83aa65c34ced56edee4cb1f47b68bc"}
Oct 31 02:14:12.864783 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.864717    4476 scope.go:122] "RemoveContainer" containerID="de0a692e3e09cccb263520487ccb5d3ca80a23ad26d39185f4619252b0c6a85c"
Oct 31 02:14:12.866056 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.866031    4476 generic.go:356] "Generic (PLEG): container finished" podID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerID="e757081dce247a93711fa19a9d3fa20f1b8e72c6ab70d3fd5d2338e824e95083" exitCode=0
Oct 31 02:14:12.866208 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.866086    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"
Oct 31 02:14:12.866208 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.866106    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" event={"ID":"3b2d11a3-2014-4d3c-b1e6-c98233a16c74","Type":"ContainerDied","Data":"e757081dce247a93711fa19a9d3fa20f1b8e72c6ab70d3fd5d2338e824e95083"}
Oct 31 02:14:12.866208 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.866131    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" event={"ID":"3b2d11a3-2014-4d3c-b1e6-c98233a16c74","Type":"ContainerDied","Data":"7770ff8827139658923bde9db307e3f39e374418b038ea50e0dba14eec6ce497"}
Oct 31 02:14:12.873375 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.873349    4476 scope.go:122] "RemoveContainer" containerID="de0a692e3e09cccb263520487ccb5d3ca80a23ad26d39185f4619252b0c6a85c"
Oct 31 02:14:12.873632 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:14:12.873611    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0a692e3e09cccb263520487ccb5d3ca80a23ad26d39185f4619252b0c6a85c\": container with ID starting with de0a692e3e09cccb263520487ccb5d3ca80a23ad26d39185f4619252b0c6a85c not found: ID does not exist" containerID="de0a692e3e09cccb263520487ccb5d3ca80a23ad26d39185f4619252b0c6a85c"
Oct 31 02:14:12.873733 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.873637    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0a692e3e09cccb263520487ccb5d3ca80a23ad26d39185f4619252b0c6a85c"} err="failed to get container status \"de0a692e3e09cccb263520487ccb5d3ca80a23ad26d39185f4619252b0c6a85c\": rpc error: code = NotFound desc = could not find container \"de0a692e3e09cccb263520487ccb5d3ca80a23ad26d39185f4619252b0c6a85c\": container with ID starting with de0a692e3e09cccb263520487ccb5d3ca80a23ad26d39185f4619252b0c6a85c not found: ID does not exist"
Oct 31 02:14:12.873733 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.873656    4476 scope.go:122] "RemoveContainer" containerID="e757081dce247a93711fa19a9d3fa20f1b8e72c6ab70d3fd5d2338e824e95083"
Oct 31 02:14:12.881750 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.881732    4476 scope.go:122] "RemoveContainer" containerID="e757081dce247a93711fa19a9d3fa20f1b8e72c6ab70d3fd5d2338e824e95083"
Oct 31 02:14:12.882006 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:14:12.881984    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e757081dce247a93711fa19a9d3fa20f1b8e72c6ab70d3fd5d2338e824e95083\": container with ID starting with e757081dce247a93711fa19a9d3fa20f1b8e72c6ab70d3fd5d2338e824e95083 not found: ID does not exist" containerID="e757081dce247a93711fa19a9d3fa20f1b8e72c6ab70d3fd5d2338e824e95083"
Oct 31 02:14:12.882062 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:12.882012    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e757081dce247a93711fa19a9d3fa20f1b8e72c6ab70d3fd5d2338e824e95083"} err="failed to get container status \"e757081dce247a93711fa19a9d3fa20f1b8e72c6ab70d3fd5d2338e824e95083\": rpc error: code = NotFound desc = could not find container \"e757081dce247a93711fa19a9d3fa20f1b8e72c6ab70d3fd5d2338e824e95083\": container with ID starting with e757081dce247a93711fa19a9d3fa20f1b8e72c6ab70d3fd5d2338e824e95083 not found: ID does not exist"
Oct 31 02:14:13.026571 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:13.026539    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"]
Oct 31 02:14:13.030284 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:13.030261    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq"]
Oct 31 02:14:13.139103 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:13.139021    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"]
Oct 31 02:14:13.140859 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:13.140840    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/distinct-2-w4-acd385-kserve-66b5f46688-gb9bx"]
Oct 31 02:14:13.395157 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:13.395014    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:13.458533 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:13.458496    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/premium-w4-acd385-kserve-65676959d8-579fq" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:14:14.452469 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:14.452427    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" path="/var/lib/kubelet/pods/3b2d11a3-2014-4d3c-b1e6-c98233a16c74/volumes"
Oct 31 02:14:14.453005 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:14.452983    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" path="/var/lib/kubelet/pods/4a67d783-3ecc-4425-8e50-8b1e72f4d57b/volumes"
Oct 31 02:14:18.407682 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.407650    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"]
Oct 31 02:14:18.408210 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.407849    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c" podUID="66b467a1-84b0-4f07-acda-74bf7ccb1be2" containerName="maas-api" containerID="cri-o://344ce5a99fe87c08ecfb21b8b79162a980f09e5c04e369df05eabdc7c2e58187" gracePeriod=30
Oct 31 02:14:18.544844 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.544797    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:18.660287 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.660237    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 02:14:18.746522 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.746498    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-maas-api-tls\") pod \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") "
Oct 31 02:14:18.746680 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.746565    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-metrics-certs\") pod \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") "
Oct 31 02:14:18.746680 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.746622    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b467a1-84b0-4f07-acda-74bf7ccb1be2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b467a1-84b0-4f07-acda-74bf7ccb1be2-openshift-service-ca\") pod \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") "
Oct 31 02:14:18.746680 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.746654    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b467a1-84b0-4f07-acda-74bf7ccb1be2-kube-api-access-gcsm7\" (UniqueName: \"kubernetes.io/projected/66b467a1-84b0-4f07-acda-74bf7ccb1be2-kube-api-access-gcsm7\") pod \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\" (UID: \"66b467a1-84b0-4f07-acda-74bf7ccb1be2\") "
Oct 31 02:14:18.746978 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.746951    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b467a1-84b0-4f07-acda-74bf7ccb1be2-openshift-service-ca" pod "66b467a1-84b0-4f07-acda-74bf7ccb1be2" (UID: "66b467a1-84b0-4f07-acda-74bf7ccb1be2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:18.748851 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.748826    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-metrics-certs" pod "66b467a1-84b0-4f07-acda-74bf7ccb1be2" (UID: "66b467a1-84b0-4f07-acda-74bf7ccb1be2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:14:18.749411 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.749387    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b467a1-84b0-4f07-acda-74bf7ccb1be2-kube-api-access-gcsm7" pod "66b467a1-84b0-4f07-acda-74bf7ccb1be2" (UID: "66b467a1-84b0-4f07-acda-74bf7ccb1be2"). InnerVolumeSpecName "kube-api-access-gcsm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:18.749411 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.749399    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-maas-api-tls" pod "66b467a1-84b0-4f07-acda-74bf7ccb1be2" (UID: "66b467a1-84b0-4f07-acda-74bf7ccb1be2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:14:18.847549 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.847520    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-metrics-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:18.847549 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.847544    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b467a1-84b0-4f07-acda-74bf7ccb1be2-openshift-service-ca\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:18.847549 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.847557    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcsm7\" (UniqueName: \"kubernetes.io/projected/66b467a1-84b0-4f07-acda-74bf7ccb1be2-kube-api-access-gcsm7\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:18.847779 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.847566    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b467a1-84b0-4f07-acda-74bf7ccb1be2-maas-api-tls\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:18.895366 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.895333    4476 generic.go:356] "Generic (PLEG): container finished" podID="66b467a1-84b0-4f07-acda-74bf7ccb1be2" containerID="344ce5a99fe87c08ecfb21b8b79162a980f09e5c04e369df05eabdc7c2e58187" exitCode=0
Oct 31 02:14:18.895507 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.895391    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"
Oct 31 02:14:18.895507 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.895427    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c" event={"ID":"66b467a1-84b0-4f07-acda-74bf7ccb1be2","Type":"ContainerDied","Data":"344ce5a99fe87c08ecfb21b8b79162a980f09e5c04e369df05eabdc7c2e58187"}
Oct 31 02:14:18.895507 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.895480    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c" event={"ID":"66b467a1-84b0-4f07-acda-74bf7ccb1be2","Type":"ContainerDied","Data":"e783ee9a0de949d5bce153168d5bb065bb4e1bc61a20b9e5631b1af70b3be1e3"}
Oct 31 02:14:18.895507 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.895504    4476 scope.go:122] "RemoveContainer" containerID="344ce5a99fe87c08ecfb21b8b79162a980f09e5c04e369df05eabdc7c2e58187"
Oct 31 02:14:18.905582 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.905561    4476 scope.go:122] "RemoveContainer" containerID="344ce5a99fe87c08ecfb21b8b79162a980f09e5c04e369df05eabdc7c2e58187"
Oct 31 02:14:18.905854 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:14:18.905834    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344ce5a99fe87c08ecfb21b8b79162a980f09e5c04e369df05eabdc7c2e58187\": container with ID starting with 344ce5a99fe87c08ecfb21b8b79162a980f09e5c04e369df05eabdc7c2e58187 not found: ID does not exist" containerID="344ce5a99fe87c08ecfb21b8b79162a980f09e5c04e369df05eabdc7c2e58187"
Oct 31 02:14:18.905910 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:18.905863    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344ce5a99fe87c08ecfb21b8b79162a980f09e5c04e369df05eabdc7c2e58187"} err="failed to get container status \"344ce5a99fe87c08ecfb21b8b79162a980f09e5c04e369df05eabdc7c2e58187\": rpc error: code = NotFound desc = could not find container \"344ce5a99fe87c08ecfb21b8b79162a980f09e5c04e369df05eabdc7c2e58187\": container with ID starting with 344ce5a99fe87c08ecfb21b8b79162a980f09e5c04e369df05eabdc7c2e58187 not found: ID does not exist"
Oct 31 02:14:19.064624 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:19.064591    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"]
Oct 31 02:14:19.066102 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:19.066082    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-acd385-79d758567-h2n7c"]
Oct 31 02:14:20.044616 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:20.044564    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:20.331359 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:20.331276    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:20.452991 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:20.452965    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b467a1-84b0-4f07-acda-74bf7ccb1be2" path="/var/lib/kubelet/pods/66b467a1-84b0-4f07-acda-74bf7ccb1be2/volumes"
Oct 31 02:14:20.960784 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:20.960756    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:27.464090 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:27.464058    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:27.647341 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:27.647307    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:27.778410 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:27.778386    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"]
Oct 31 02:14:27.778609 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:27.778589    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk" podUID="b5b406da-fb74-4089-914d-569cc542526b" containerName="istio-proxy" containerID="cri-o://0351f49235332d2410549ea6d305286b5639ce383d9330aeb2dca4d269e23353" gracePeriod=30
Oct 31 02:14:27.841294 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:27.841268    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:28.867098 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:28.867074    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f5785569d-zz4s7_14bf16ae-6e30-4983-8f4d-af34c4dca0eb/authorino/0.log"
Oct 31 02:14:29.180798 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:14:29.180701    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:14:31.458955 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:31.458911    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:14:31.459381 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:31.458990    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk" podUID="b5b406da-fb74-4089-914d-569cc542526b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:14:32.962666 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:32.962633    4476 generic.go:356] "Generic (PLEG): container finished" podID="b5b406da-fb74-4089-914d-569cc542526b" containerID="0351f49235332d2410549ea6d305286b5639ce383d9330aeb2dca4d269e23353" exitCode=0
Oct 31 02:14:32.963031 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:32.962708    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk" event={"ID":"b5b406da-fb74-4089-914d-569cc542526b","Type":"ContainerDied","Data":"0351f49235332d2410549ea6d305286b5639ce383d9330aeb2dca4d269e23353"}
Oct 31 02:14:33.022395 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.022369    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 02:14:33.171003 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.170935    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-workload-socket\") pod \"b5b406da-fb74-4089-914d-569cc542526b\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") "
Oct 31 02:14:33.171003 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.170977    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b406da-fb74-4089-914d-569cc542526b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b406da-fb74-4089-914d-569cc542526b-istiod-ca-cert\") pod \"b5b406da-fb74-4089-914d-569cc542526b\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") "
Oct 31 02:14:33.171003 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.170995    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-workload-certs\") pod \"b5b406da-fb74-4089-914d-569cc542526b\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") "
Oct 31 02:14:33.171317 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.171034    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5b406da-fb74-4089-914d-569cc542526b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b406da-fb74-4089-914d-569cc542526b-istio-podinfo\") pod \"b5b406da-fb74-4089-914d-569cc542526b\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") "
Oct 31 02:14:33.171317 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.171066    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b406da-fb74-4089-914d-569cc542526b-kube-api-access-fh4lp\" (UniqueName: \"kubernetes.io/projected/b5b406da-fb74-4089-914d-569cc542526b-kube-api-access-fh4lp\") pod \"b5b406da-fb74-4089-914d-569cc542526b\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") "
Oct 31 02:14:33.171317 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.171096    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-istio-envoy\") pod \"b5b406da-fb74-4089-914d-569cc542526b\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") "
Oct 31 02:14:33.171317 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.171142    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b406da-fb74-4089-914d-569cc542526b-istio-token\" (UniqueName: \"kubernetes.io/projected/b5b406da-fb74-4089-914d-569cc542526b-istio-token\") pod \"b5b406da-fb74-4089-914d-569cc542526b\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") "
Oct 31 02:14:33.171317 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.171240    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-credential-socket\") pod \"b5b406da-fb74-4089-914d-569cc542526b\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") "
Oct 31 02:14:33.171317 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.171272    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-istio-data\") pod \"b5b406da-fb74-4089-914d-569cc542526b\" (UID: \"b5b406da-fb74-4089-914d-569cc542526b\") "
Oct 31 02:14:33.171624 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.171263    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-workload-socket" pod "b5b406da-fb74-4089-914d-569cc542526b" (UID: "b5b406da-fb74-4089-914d-569cc542526b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:33.171624 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.171392    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-workload-certs" pod "b5b406da-fb74-4089-914d-569cc542526b" (UID: "b5b406da-fb74-4089-914d-569cc542526b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:33.171624 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.171440    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b406da-fb74-4089-914d-569cc542526b-istiod-ca-cert" pod "b5b406da-fb74-4089-914d-569cc542526b" (UID: "b5b406da-fb74-4089-914d-569cc542526b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:33.171624 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.171524    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-credential-socket" pod "b5b406da-fb74-4089-914d-569cc542526b" (UID: "b5b406da-fb74-4089-914d-569cc542526b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:33.171797 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.171727    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-istio-data" pod "b5b406da-fb74-4089-914d-569cc542526b" (UID: "b5b406da-fb74-4089-914d-569cc542526b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:33.173546 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.173513    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-istio-envoy" pod "b5b406da-fb74-4089-914d-569cc542526b" (UID: "b5b406da-fb74-4089-914d-569cc542526b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:33.173647 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.173583    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b406da-fb74-4089-914d-569cc542526b-kube-api-access-fh4lp" pod "b5b406da-fb74-4089-914d-569cc542526b" (UID: "b5b406da-fb74-4089-914d-569cc542526b"). InnerVolumeSpecName "kube-api-access-fh4lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:33.173647 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.173583    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b406da-fb74-4089-914d-569cc542526b-istio-token" pod "b5b406da-fb74-4089-914d-569cc542526b" (UID: "b5b406da-fb74-4089-914d-569cc542526b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:33.173647 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.173609    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5b406da-fb74-4089-914d-569cc542526b-istio-podinfo" pod "b5b406da-fb74-4089-914d-569cc542526b" (UID: "b5b406da-fb74-4089-914d-569cc542526b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:14:33.272482 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.272453    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-workload-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:33.272482 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.272479    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b406da-fb74-4089-914d-569cc542526b-istiod-ca-cert\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:33.272482 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.272488    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-workload-certs\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:33.272692 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.272496    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b406da-fb74-4089-914d-569cc542526b-istio-podinfo\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:33.272692 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.272506    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh4lp\" (UniqueName: \"kubernetes.io/projected/b5b406da-fb74-4089-914d-569cc542526b-kube-api-access-fh4lp\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:33.272692 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.272515    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-istio-envoy\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:33.272692 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.272523    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b406da-fb74-4089-914d-569cc542526b-istio-token\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:33.272692 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.272531    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-credential-socket\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:33.272692 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.272538    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b406da-fb74-4089-914d-569cc542526b-istio-data\") on node \"ip-10-0-139-174.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:14:33.969546 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.969508    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"
Oct 31 02:14:33.969992 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.969508    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk" event={"ID":"b5b406da-fb74-4089-914d-569cc542526b","Type":"ContainerDied","Data":"f993665fca3de5bdacbd55066c943d57029cf33f7d7227a67816b3547d574709"}
Oct 31 02:14:33.969992 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:33.969647    4476 scope.go:122] "RemoveContainer" containerID="0351f49235332d2410549ea6d305286b5639ce383d9330aeb2dca4d269e23353"
Oct 31 02:14:34.166368 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:34.166342    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:34.181014 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:34.180980    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"]
Oct 31 02:14:34.231804 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:34.231722    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4p6zk"]
Oct 31 02:14:34.361914 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:34.361881    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:34.453347 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:34.453319    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b406da-fb74-4089-914d-569cc542526b" path="/var/lib/kubelet/pods/b5b406da-fb74-4089-914d-569cc542526b/volumes"
Oct 31 02:14:34.541714 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:34.541668    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:34.744634 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:34.744597    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:40.318990 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:40.318936    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:43.139114 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:43.139049    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bb949dc9c-x6t7p_e766f760-98d7-44a0-978e-cf414a9e23b5/manager/0.log"
Oct 31 02:14:45.404968 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:45.404936    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:45.688452 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:45.688379    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kzt5t_9787be82-26ee-4d1c-ad0c-7d2de2f3e065/proxy/0.log"
Oct 31 02:14:46.590359 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:46.590333    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f5785569d-zz4s7_14bf16ae-6e30-4983-8f4d-af34c4dca0eb/authorino/0.log"
Oct 31 02:14:57.381143 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:14:57.381108    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bb949dc9c-x6t7p_e766f760-98d7-44a0-978e-cf414a9e23b5/manager/0.log"
Oct 31 02:14:59.181186 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:14:59.181137    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:15:10.090279 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:10.090246    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kzt5t_9787be82-26ee-4d1c-ad0c-7d2de2f3e065/proxy/0.log"
Oct 31 02:15:11.056633 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:11.056555    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-555758456c-mm88j_67afcaea-e90e-4616-8dc1-97ecd945c795/maas-api/0.log"
Oct 31 02:15:15.397690 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:15.397658    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f5785569d-zz4s7_14bf16ae-6e30-4983-8f4d-af34c4dca0eb/authorino/0.log"
Oct 31 02:15:19.508866 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:19.508833    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58c8f9d5-2hkws_32e2d8b2-67ec-4f33-abe2-b5e27b1761a9/kube-auth-proxy/0.log"
Oct 31 02:15:19.963094 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:19.963063    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-gbwwl_b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8/istio-proxy/0.log"
Oct 31 02:15:20.559038 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:20.559008    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dmjmv_a7ab6ca9-406f-471c-a4e4-1cc24dc493cb/payload-pre-processing/0.log"
Oct 31 02:15:20.786807 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:20.786780    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vl9ww_c0881ec8-1f45-459a-b40c-29e2d2246ea0/payload-processing/0.log"
Oct 31 02:15:21.971682 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:21.971657    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-h6wb8_0b791ac7-2890-4aef-8a04-c977fd48ada1/main/0.log"
Oct 31 02:15:22.252465 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:22.252389    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-xnjmn_076bf998-c29d-4915-ba9d-a653b8185e75/main/0.log"
Oct 31 02:15:22.493779 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:22.493753    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-zclgc_6f0b68e0-1657-45d3-b42f-573f61a52ac2/main/0.log"
Oct 31 02:15:22.736096 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:22.736071    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-w8xgw_ec32f101-2d93-4344-9497-37200971b9ad/main/0.log"
Oct 31 02:15:23.191783 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:23.191756    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-9xsbq_982a3abb-abe9-4661-a517-06cc92cb966f/main/0.log"
Oct 31 02:15:23.651240 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:23.651212    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-5q9mq_48826212-3a95-460e-9e5e-91ef131c528b/main/0.log"
Oct 31 02:15:26.345346 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:26.345312    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:15:27.020804 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:27.020774    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:15:29.180530 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:15:29.180486    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:15:43.688547 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:43.688517    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:15:43.855108 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:43.855080    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:15:45.567972 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:15:45.567941    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:15:59.180310 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:15:59.180273    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:16:04.841481 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841446    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5dndx/must-gather-v4nh5"]
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841788    4476 cpu_manager.go:414] "containerMap: removing container" podUID="3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841799    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841812    4476 cpu_manager.go:414] "containerMap: removing container" podUID="dcdec68c-a09e-45ac-8ed8-1aa397ccf660" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841817    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcdec68c-a09e-45ac-8ed8-1aa397ccf660" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841826    4476 cpu_manager.go:414] "containerMap: removing container" podUID="fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841830    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841836    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c382414a-6f3f-41fc-b773-b315da186244" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841840    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c382414a-6f3f-41fc-b773-b315da186244" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841846    4476 cpu_manager.go:414] "containerMap: removing container" podUID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841851    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841858    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b5b406da-fb74-4089-914d-569cc542526b" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841862    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b406da-fb74-4089-914d-569cc542526b" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841870    4476 cpu_manager.go:414] "containerMap: removing container" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841874    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841880    4476 cpu_manager.go:414] "containerMap: removing container" podUID="6b639316-fe8e-40e3-bab9-19c4e66f9796" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841885    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b639316-fe8e-40e3-bab9-19c4e66f9796" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841891    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841895    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841902    4476 cpu_manager.go:414] "containerMap: removing container" podUID="6d3fd6dd-6237-4bc8-a87d-2ccc8d515839" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841906    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3fd6dd-6237-4bc8-a87d-2ccc8d515839" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841916    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841921    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841927    4476 cpu_manager.go:414] "containerMap: removing container" podUID="69028e29-6686-4cd4-bf5a-94446d2c70d2" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841931    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69028e29-6686-4cd4-bf5a-94446d2c70d2" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841937    4476 cpu_manager.go:414] "containerMap: removing container" podUID="972f6d48-cd71-4ccf-bc10-192462149181" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841942    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972f6d48-cd71-4ccf-bc10-192462149181" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841948    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841952    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841960    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841964    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841974    4476 cpu_manager.go:414] "containerMap: removing container" podUID="a7b29e19-6e82-4c47-a9f6-d66f60afd3ca" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841978    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b29e19-6e82-4c47-a9f6-d66f60afd3ca" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841987    4476 cpu_manager.go:414] "containerMap: removing container" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841992    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.841997    4476 cpu_manager.go:414] "containerMap: removing container" podUID="047a012c-e995-469b-a04a-ce12cdd56f94" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842002    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047a012c-e995-469b-a04a-ce12cdd56f94" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842008    4476 cpu_manager.go:414] "containerMap: removing container" podUID="810203c9-65fd-4958-a46e-7ecf24a22f41" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842013    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810203c9-65fd-4958-a46e-7ecf24a22f41" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842020    4476 cpu_manager.go:414] "containerMap: removing container" podUID="3199cbdc-b6ba-4212-af9d-fd3d42a6f944" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842024    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3199cbdc-b6ba-4212-af9d-fd3d42a6f944" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842031    4476 cpu_manager.go:414] "containerMap: removing container" podUID="27377642-d40d-4498-af5d-16ca6a58ef54" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842036    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27377642-d40d-4498-af5d-16ca6a58ef54" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842041    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b14aee9c-3186-4e2a-9c84-9b50cb844194" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842045    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b14aee9c-3186-4e2a-9c84-9b50cb844194" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842052    4476 cpu_manager.go:414] "containerMap: removing container" podUID="6af72007-50de-4d45-9c18-bdea122f32e6" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842056    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af72007-50de-4d45-9c18-bdea122f32e6" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842061    4476 cpu_manager.go:414] "containerMap: removing container" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842065    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842071    4476 cpu_manager.go:414] "containerMap: removing container" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842075    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842081    4476 cpu_manager.go:414] "containerMap: removing container" podUID="66b467a1-84b0-4f07-acda-74bf7ccb1be2" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842085    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b467a1-84b0-4f07-acda-74bf7ccb1be2" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842090    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c76cc5c8-d8e3-44d4-b419-9a38439489e2" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842096    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76cc5c8-d8e3-44d4-b419-9a38439489e2" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842103    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842108    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842115    4476 cpu_manager.go:414] "containerMap: removing container" podUID="9dcbe11f-bf2c-4e64-91db-96868f1f38ea" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842119    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dcbe11f-bf2c-4e64-91db-96868f1f38ea" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842125    4476 cpu_manager.go:414] "containerMap: removing container" podUID="0beaabcf-d87c-4769-9a19-84e9124568a3" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842130    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0beaabcf-d87c-4769-9a19-84e9124568a3" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842138    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ef906818-75f2-40e3-a4b1-16f6f5e40a70" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842142    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef906818-75f2-40e3-a4b1-16f6f5e40a70" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842147    4476 cpu_manager.go:414] "containerMap: removing container" podUID="cbfffc20-e86c-42da-b9d2-58d24d9152aa" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842151    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbfffc20-e86c-42da-b9d2-58d24d9152aa" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842156    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5efb0ead-026d-472a-b36d-8ad0cccd3487" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842160    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5efb0ead-026d-472a-b36d-8ad0cccd3487" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842166    4476 cpu_manager.go:414] "containerMap: removing container" podUID="eb736102-7fdb-47d2-93f9-fa3fa5d67ed5" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842170    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb736102-7fdb-47d2-93f9-fa3fa5d67ed5" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842189    4476 cpu_manager.go:414] "containerMap: removing container" podUID="4ea835fa-fab8-4d8b-9b94-7c2478ad41e9" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842194    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea835fa-fab8-4d8b-9b94-7c2478ad41e9" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842203    4476 cpu_manager.go:414] "containerMap: removing container" podUID="3734f793-a26d-4f0b-8f5a-954b10bb51c7" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842207    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3734f793-a26d-4f0b-8f5a-954b10bb51c7" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842213    4476 cpu_manager.go:414] "containerMap: removing container" podUID="762230e3-0692-4a5b-8e29-13bf8a3a4b3c" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842217    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762230e3-0692-4a5b-8e29-13bf8a3a4b3c" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842223    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ae3e94c6-c7e3-4e24-a832-4e632806f191" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842228    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3e94c6-c7e3-4e24-a832-4e632806f191" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842278    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842285    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842294    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b29e19-6e82-4c47-a9f6-d66f60afd3ca" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842299    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2d11a3-2014-4d3c-b1e6-c98233a16c74" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842305    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="047a012c-e995-469b-a04a-ce12cdd56f94" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842310    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="810203c9-65fd-4958-a46e-7ecf24a22f41" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842316    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3199cbdc-b6ba-4212-af9d-fd3d42a6f944" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842322    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="27377642-d40d-4498-af5d-16ca6a58ef54" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842327    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842333    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b14aee9c-3186-4e2a-9c84-9b50cb844194" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842339    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af72007-50de-4d45-9c18-bdea122f32e6" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842344    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbdc50dd-29d0-4610-a361-cfd11ad1f501" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842349    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="912dfe65-0c1e-4445-9d00-cf197ad4a93b" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842354    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b467a1-84b0-4f07-acda-74bf7ccb1be2" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842359    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76cc5c8-d8e3-44d4-b419-9a38439489e2" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842363    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc73dbe-90f0-4dc0-9ce9-c1e5cec12912" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842368    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dcbe11f-bf2c-4e64-91db-96868f1f38ea" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842374    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0beaabcf-d87c-4769-9a19-84e9124568a3" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842380    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef906818-75f2-40e3-a4b1-16f6f5e40a70" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842385    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbfffc20-e86c-42da-b9d2-58d24d9152aa" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842389    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5efb0ead-026d-472a-b36d-8ad0cccd3487" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842393    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb736102-7fdb-47d2-93f9-fa3fa5d67ed5" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842399    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea835fa-fab8-4d8b-9b94-7c2478ad41e9" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842405    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3734f793-a26d-4f0b-8f5a-954b10bb51c7" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842409    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="762230e3-0692-4a5b-8e29-13bf8a3a4b3c" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842413    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3e94c6-c7e3-4e24-a832-4e632806f191" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842418    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a3a5819-7b4f-4177-9a2b-5cd76e2a7c14" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842423    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcdec68c-a09e-45ac-8ed8-1aa397ccf660" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842429    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffaf3f9-2e8d-4aa1-a5d1-c3db5f0f5194" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842433    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c382414a-6f3f-41fc-b773-b315da186244" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842438    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842442    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b406da-fb74-4089-914d-569cc542526b" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842446    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a67d783-3ecc-4425-8e50-8b1e72f4d57b" containerName="main"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842451    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b639316-fe8e-40e3-bab9-19c4e66f9796" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842456    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842461    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3fd6dd-6237-4bc8-a87d-2ccc8d515839" containerName="payload-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842466    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="69028e29-6686-4cd4-bf5a-94446d2c70d2" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842471    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="972f6d48-cd71-4ccf-bc10-192462149181" containerName="payload-pre-processing"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842516    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842520    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842543    4476 cpu_manager.go:414] "containerMap: removing container" podUID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842547    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069276a8-4b8e-4db0-9dd1-86c7b402309d" containerName="istio-proxy"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842594    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26dc7e6-eb1f-4733-930f-836d9703d440" containerName="maas-api"
Oct 31 02:16:04.843821 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.842601    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9d6d08-fc6a-4c3d-8ae3-6bed8194d2b5" containerName="maas-api"
Oct 31 02:16:04.847995 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.845317    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5dndx/must-gather-v4nh5"
Oct 31 02:16:04.903302 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.903273    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5dndx/must-gather-v4nh5"]
Oct 31 02:16:04.903892 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.903874    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:16:04.950238 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.950208    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5dndx\"/\"openshift-service-ca.crt\""
Oct 31 02:16:04.950454 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.950437    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5dndx\"/\"kube-root-ca.crt\""
Oct 31 02:16:04.950530 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.950512    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5dndx\"/\"default-dockercfg-wd6kq\""
Oct 31 02:16:04.958769 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.958739    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kz6s\" (UniqueName: \"kubernetes.io/projected/ee0979fb-c121-438a-a511-bb1289738175-kube-api-access-8kz6s\") pod \"must-gather-v4nh5\" (UID: \"ee0979fb-c121-438a-a511-bb1289738175\") " pod="openshift-must-gather-5dndx/must-gather-v4nh5"
Oct 31 02:16:04.958890 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:04.958856    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ee0979fb-c121-438a-a511-bb1289738175-must-gather-output\") pod \"must-gather-v4nh5\" (UID: \"ee0979fb-c121-438a-a511-bb1289738175\") " pod="openshift-must-gather-5dndx/must-gather-v4nh5"
Oct 31 02:16:05.059304 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:05.059275    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ee0979fb-c121-438a-a511-bb1289738175-must-gather-output\") pod \"must-gather-v4nh5\" (UID: \"ee0979fb-c121-438a-a511-bb1289738175\") " pod="openshift-must-gather-5dndx/must-gather-v4nh5"
Oct 31 02:16:05.059463 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:05.059317    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kz6s\" (UniqueName: \"kubernetes.io/projected/ee0979fb-c121-438a-a511-bb1289738175-kube-api-access-8kz6s\") pod \"must-gather-v4nh5\" (UID: \"ee0979fb-c121-438a-a511-bb1289738175\") " pod="openshift-must-gather-5dndx/must-gather-v4nh5"
Oct 31 02:16:05.059687 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:05.059665    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ee0979fb-c121-438a-a511-bb1289738175-must-gather-output\") pod \"must-gather-v4nh5\" (UID: \"ee0979fb-c121-438a-a511-bb1289738175\") " pod="openshift-must-gather-5dndx/must-gather-v4nh5"
Oct 31 02:16:05.120601 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:05.120533    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kz6s\" (UniqueName: \"kubernetes.io/projected/ee0979fb-c121-438a-a511-bb1289738175-kube-api-access-8kz6s\") pod \"must-gather-v4nh5\" (UID: \"ee0979fb-c121-438a-a511-bb1289738175\") " pod="openshift-must-gather-5dndx/must-gather-v4nh5"
Oct 31 02:16:05.158170 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:05.158144    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5dndx/must-gather-v4nh5"
Oct 31 02:16:05.528454 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:05.528429    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5dndx/must-gather-v4nh5"]
Oct 31 02:16:05.531120 ip-10-0-139-174 kubenswrapper[4476]: W1031 02:16:05.531090    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee0979fb_c121_438a_a511_bb1289738175.slice/crio-aff44157c5d979eadd661d2a57becbb716d6dc06578420e50c24c8e91d212cb9 WatchSource:0}: Error finding container aff44157c5d979eadd661d2a57becbb716d6dc06578420e50c24c8e91d212cb9: Status 404 returned error can't find the container with id aff44157c5d979eadd661d2a57becbb716d6dc06578420e50c24c8e91d212cb9
Oct 31 02:16:05.532937 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:05.532920    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 02:16:06.375065 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:06.375028    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5dndx/must-gather-v4nh5" event={"ID":"ee0979fb-c121-438a-a511-bb1289738175","Type":"ContainerStarted","Data":"aff44157c5d979eadd661d2a57becbb716d6dc06578420e50c24c8e91d212cb9"}
Oct 31 02:16:06.391266 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:06.391241    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:16:07.381229 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:07.381124    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5dndx/must-gather-v4nh5" event={"ID":"ee0979fb-c121-438a-a511-bb1289738175","Type":"ContainerStarted","Data":"7a45c9d7956c63f15ae158c4d840efbdb5e9f4c17b69c52e62f0dba25b6667e6"}
Oct 31 02:16:07.381229 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:07.381192    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5dndx/must-gather-v4nh5" event={"ID":"ee0979fb-c121-438a-a511-bb1289738175","Type":"ContainerStarted","Data":"dff0532c10d191d7500a27382a45396b6239162a283a71a0e565ce09c8200d1f"}
Oct 31 02:16:07.496847 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:07.496790    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5dndx/must-gather-v4nh5" podStartSLOduration=2.184443885 podStartE2EDuration="3.496776078s" totalImagesPullingTime="1.312332193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:16:07.49461707 +0000 UTC m=+4777.184500198" watchObservedRunningTime="2026-10-31 02:16:07.496776078 +0000 UTC m=+4777.186659205"
Oct 31 02:16:14.820696 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:14.820656    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w5hdq/must-gather-9p6n9"]
Oct 31 02:16:14.825927 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:14.825904    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5hdq/must-gather-9p6n9"
Oct 31 02:16:14.858591 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:14.858565    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jdmsp_e0cade88-4da4-467c-9656-cce6afe1dd62/global-pull-secret-syncer/0.log"
Oct 31 02:16:14.886532 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:14.886508    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:16:14.930235 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:14.930211    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w5hdq\"/\"openshift-service-ca.crt\""
Oct 31 02:16:14.930235 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:14.930243    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w5hdq\"/\"kube-root-ca.crt\""
Oct 31 02:16:14.930427 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:14.930278    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w5hdq\"/\"default-dockercfg-twf7v\""
Oct 31 02:16:14.932086 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:14.932066    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5hdq/must-gather-9p6n9"]
Oct 31 02:16:14.951784 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:14.951760    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc9rf\" (UniqueName: \"kubernetes.io/projected/98cd8583-2d83-4dd6-bd69-1c53170f0c83-kube-api-access-wc9rf\") pod \"must-gather-9p6n9\" (UID: \"98cd8583-2d83-4dd6-bd69-1c53170f0c83\") " pod="openshift-must-gather-w5hdq/must-gather-9p6n9"
Oct 31 02:16:14.951913 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:14.951891    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/98cd8583-2d83-4dd6-bd69-1c53170f0c83-must-gather-output\") pod \"must-gather-9p6n9\" (UID: \"98cd8583-2d83-4dd6-bd69-1c53170f0c83\") " pod="openshift-must-gather-w5hdq/must-gather-9p6n9"
Oct 31 02:16:15.052661 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:15.052630    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/98cd8583-2d83-4dd6-bd69-1c53170f0c83-must-gather-output\") pod \"must-gather-9p6n9\" (UID: \"98cd8583-2d83-4dd6-bd69-1c53170f0c83\") " pod="openshift-must-gather-w5hdq/must-gather-9p6n9"
Oct 31 02:16:15.052795 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:15.052671    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc9rf\" (UniqueName: \"kubernetes.io/projected/98cd8583-2d83-4dd6-bd69-1c53170f0c83-kube-api-access-wc9rf\") pod \"must-gather-9p6n9\" (UID: \"98cd8583-2d83-4dd6-bd69-1c53170f0c83\") " pod="openshift-must-gather-w5hdq/must-gather-9p6n9"
Oct 31 02:16:15.052981 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:15.052963    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/98cd8583-2d83-4dd6-bd69-1c53170f0c83-must-gather-output\") pod \"must-gather-9p6n9\" (UID: \"98cd8583-2d83-4dd6-bd69-1c53170f0c83\") " pod="openshift-must-gather-w5hdq/must-gather-9p6n9"
Oct 31 02:16:15.109552 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:15.109490    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc9rf\" (UniqueName: \"kubernetes.io/projected/98cd8583-2d83-4dd6-bd69-1c53170f0c83-kube-api-access-wc9rf\") pod \"must-gather-9p6n9\" (UID: \"98cd8583-2d83-4dd6-bd69-1c53170f0c83\") " pod="openshift-must-gather-w5hdq/must-gather-9p6n9"
Oct 31 02:16:15.139152 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:15.139129    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5hdq/must-gather-9p6n9"
Oct 31 02:16:15.191389 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:15.191327    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:16:15.509133 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:15.509111    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5hdq/must-gather-9p6n9"]
Oct 31 02:16:15.511200 ip-10-0-139-174 kubenswrapper[4476]: W1031 02:16:15.511158    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98cd8583_2d83_4dd6_bd69_1c53170f0c83.slice/crio-862d1e9185b2d6b1c33ba6cc58d35940f84e0f14b29b48b2348eebd5fccfe46f WatchSource:0}: Error finding container 862d1e9185b2d6b1c33ba6cc58d35940f84e0f14b29b48b2348eebd5fccfe46f: Status 404 returned error can't find the container with id 862d1e9185b2d6b1c33ba6cc58d35940f84e0f14b29b48b2348eebd5fccfe46f
Oct 31 02:16:16.421995 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:16.421957    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5hdq/must-gather-9p6n9" event={"ID":"98cd8583-2d83-4dd6-bd69-1c53170f0c83","Type":"ContainerStarted","Data":"862d1e9185b2d6b1c33ba6cc58d35940f84e0f14b29b48b2348eebd5fccfe46f"}
Oct 31 02:16:16.746871 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:16.746798    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log"
Oct 31 02:16:19.241086 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:19.241061    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal_7fc072a2ae8e85ec3464a94d2bac4401/haproxy/0.log"
Oct 31 02:16:22.467861 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:22.467827    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5hdq/must-gather-9p6n9" event={"ID":"98cd8583-2d83-4dd6-bd69-1c53170f0c83","Type":"ContainerStarted","Data":"f326e2e93f1eea255b11a325e4036da1736c334793d178bb404a75e1ae692455"}
Oct 31 02:16:22.467861 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:22.467867    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5hdq/must-gather-9p6n9" event={"ID":"98cd8583-2d83-4dd6-bd69-1c53170f0c83","Type":"ContainerStarted","Data":"69055c544565c7f285350144682d0cf714b2be67f644f2c7296d6733ec394f1e"}
Oct 31 02:16:22.580165 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:22.580088    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-w5hdq/must-gather-9p6n9" podStartSLOduration=2.407893749 podStartE2EDuration="8.580067651s" totalImagesPullingTime="6.172173902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:16:22.57934982 +0000 UTC m=+4792.269232948" watchObservedRunningTime="2026-10-31 02:16:22.580067651 +0000 UTC m=+4792.269950781"
Oct 31 02:16:25.643993 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:25.643959    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bb949dc9c-x6t7p_e766f760-98d7-44a0-978e-cf414a9e23b5/manager/0.log"
Oct 31 02:16:25.822279 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:25.822249    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bb949dc9c-x6t7p_e766f760-98d7-44a0-978e-cf414a9e23b5/manager/0.log"
Oct 31 02:16:28.905023 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:28.904993    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kzt5t_9787be82-26ee-4d1c-ad0c-7d2de2f3e065/proxy/0.log"
Oct 31 02:16:29.000867 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:29.000841    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kzt5t_9787be82-26ee-4d1c-ad0c-7d2de2f3e065/proxy/0.log"
Oct 31 02:16:29.196352 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:16:29.196274    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:16:31.076380 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:31.076352    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 02:16:31.082219 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:31.082169    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wpwjb_de5a8a39-81f2-4538-99d6-6da9bc70e138/extract/0.log"
Oct 31 02:16:31.698240 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:31.698208    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bb949dc9c-x6t7p_e766f760-98d7-44a0-978e-cf414a9e23b5/manager/0.log"
Oct 31 02:16:31.785687 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:31.785662    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bb949dc9c-x6t7p_e766f760-98d7-44a0-978e-cf414a9e23b5/manager/0.log"
Oct 31 02:16:35.056653 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:35.056624    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kzt5t_9787be82-26ee-4d1c-ad0c-7d2de2f3e065/proxy/0.log"
Oct 31 02:16:35.109616 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:35.109588    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kzt5t_9787be82-26ee-4d1c-ad0c-7d2de2f3e065/proxy/0.log"
Oct 31 02:16:38.003954 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:38.003912    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bb949dc9c-x6t7p_e766f760-98d7-44a0-978e-cf414a9e23b5/manager/0.log"
Oct 31 02:16:39.507736 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:39.507700    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:16:40.841279 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:40.841198    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kzt5t_9787be82-26ee-4d1c-ad0c-7d2de2f3e065/proxy/0.log"
Oct 31 02:16:45.195198 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:45.194905    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58c8f9d5-2hkws_32e2d8b2-67ec-4f33-abe2-b5e27b1761a9/kube-auth-proxy/0.log"
Oct 31 02:16:45.642550 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:45.642517    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-gbwwl_b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8/istio-proxy/0.log"
Oct 31 02:16:45.911004 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:45.910892    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:16:46.094063 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:46.094023    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dmjmv_a7ab6ca9-406f-471c-a4e4-1cc24dc493cb/payload-pre-processing/0.log"
Oct 31 02:16:46.319198 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:46.318232    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vl9ww_c0881ec8-1f45-459a-b40c-29e2d2246ea0/payload-processing/0.log"
Oct 31 02:16:49.445300 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:49.445266    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58c8f9d5-2hkws_32e2d8b2-67ec-4f33-abe2-b5e27b1761a9/kube-auth-proxy/0.log"
Oct 31 02:16:49.882159 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:49.882126    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-gbwwl_b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8/istio-proxy/0.log"
Oct 31 02:16:50.302209 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:50.301311    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dmjmv_a7ab6ca9-406f-471c-a4e4-1cc24dc493cb/payload-pre-processing/0.log"
Oct 31 02:16:50.512728 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:50.512334    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vl9ww_c0881ec8-1f45-459a-b40c-29e2d2246ea0/payload-processing/0.log"
Oct 31 02:16:53.552196 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:53.552143    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f5785569d-zz4s7_14bf16ae-6e30-4983-8f4d-af34c4dca0eb/authorino/0.log"
Oct 31 02:16:53.681971 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:53.681918    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f5785569d-zz4s7_14bf16ae-6e30-4983-8f4d-af34c4dca0eb/authorino/0.log"
Oct 31 02:16:54.038875 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.038843    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"]
Oct 31 02:16:54.043614 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.043593    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.137714 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.137684    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"]
Oct 31 02:16:54.165315 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.165288    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd65440d-7b05-45b9-877b-e63dc5734716-lib-modules\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.165315 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.165318    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4jrd\" (UniqueName: \"kubernetes.io/projected/bd65440d-7b05-45b9-877b-e63dc5734716-kube-api-access-c4jrd\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.165578 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.165407    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd65440d-7b05-45b9-877b-e63dc5734716-sys\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.165578 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.165557    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd65440d-7b05-45b9-877b-e63dc5734716-podres\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.165684 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.165592    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd65440d-7b05-45b9-877b-e63dc5734716-proc\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.266546 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.266518    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd65440d-7b05-45b9-877b-e63dc5734716-sys\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.266714 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.266572    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd65440d-7b05-45b9-877b-e63dc5734716-podres\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.266714 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.266596    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd65440d-7b05-45b9-877b-e63dc5734716-proc\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.266714 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.266631    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd65440d-7b05-45b9-877b-e63dc5734716-lib-modules\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.266714 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.266652    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4jrd\" (UniqueName: \"kubernetes.io/projected/bd65440d-7b05-45b9-877b-e63dc5734716-kube-api-access-c4jrd\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.266714 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.266631    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd65440d-7b05-45b9-877b-e63dc5734716-sys\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.266714 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.266666    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd65440d-7b05-45b9-877b-e63dc5734716-podres\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.266714 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.266693    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd65440d-7b05-45b9-877b-e63dc5734716-proc\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.266984 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.266725    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd65440d-7b05-45b9-877b-e63dc5734716-lib-modules\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.323213 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.323124    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4jrd\" (UniqueName: \"kubernetes.io/projected/bd65440d-7b05-45b9-877b-e63dc5734716-kube-api-access-c4jrd\") pod \"perf-node-gather-daemonset-kwt72\" (UID: \"bd65440d-7b05-45b9-877b-e63dc5734716\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.355105 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.355079    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:54.725567 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:54.725540    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"]
Oct 31 02:16:54.728744 ip-10-0-139-174 kubenswrapper[4476]: W1031 02:16:54.728711    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbd65440d_7b05_45b9_877b_e63dc5734716.slice/crio-56a3360747876c89dda9b74bb71f924e81d36ca45050e1964c89aba92ee4d18d WatchSource:0}: Error finding container 56a3360747876c89dda9b74bb71f924e81d36ca45050e1964c89aba92ee4d18d: Status 404 returned error can't find the container with id 56a3360747876c89dda9b74bb71f924e81d36ca45050e1964c89aba92ee4d18d
Oct 31 02:16:55.658732 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:55.658688    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72" event={"ID":"bd65440d-7b05-45b9-877b-e63dc5734716","Type":"ContainerStarted","Data":"07856c4963fb9d0c48f94b7d4ef4ea98e02b162896331a934ecad6130b8c0780"}
Oct 31 02:16:55.659053 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:55.658798    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72" event={"ID":"bd65440d-7b05-45b9-877b-e63dc5734716","Type":"ContainerStarted","Data":"56a3360747876c89dda9b74bb71f924e81d36ca45050e1964c89aba92ee4d18d"}
Oct 31 02:16:55.659053 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:55.658931    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:16:55.777467 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:16:55.777418    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72" podStartSLOduration=2.777403951 podStartE2EDuration="2.777403951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:16:55.776745073 +0000 UTC m=+4825.466628198" watchObservedRunningTime="2026-10-31 02:16:55.777403951 +0000 UTC m=+4825.467287109"
Oct 31 02:16:59.182698 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:16:59.182664    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:17:01.676662 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:01.676637    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-kwt72"
Oct 31 02:17:05.947975 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:05.947922    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:17:06.219431 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:06.219354    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gvgm6_e087cf8c-2361-4702-9655-3ce6ee3c6f3d/node-exporter/0.log"
Oct 31 02:17:06.630540 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:06.630516    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gvgm6_e087cf8c-2361-4702-9655-3ce6ee3c6f3d/kube-rbac-proxy/0.log"
Oct 31 02:17:07.044094 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:07.044067    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gvgm6_e087cf8c-2361-4702-9655-3ce6ee3c6f3d/init-textfile/0.log"
Oct 31 02:17:13.674208 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:13.673652    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-xqkgj_75228856-5f8f-4d69-9c35-1012563b9ed1/prometheus-operator-admission-webhook/0.log"
Oct 31 02:17:14.091636 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:14.091603    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655b45697-bp2zh_7b8249f4-55b9-40a2-9d79-2265aafb9c78/telemeter-client/0.log"
Oct 31 02:17:14.504617 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:14.504536    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655b45697-bp2zh_7b8249f4-55b9-40a2-9d79-2265aafb9c78/reload/0.log"
Oct 31 02:17:14.916336 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:14.916305    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655b45697-bp2zh_7b8249f4-55b9-40a2-9d79-2265aafb9c78/kube-rbac-proxy/0.log"
Oct 31 02:17:18.014195 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:18.014142    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:17:25.709705 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:25.709676    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-78s6n_70365afb-44c6-4aa3-8512-0f01c05ead6e/download-server/0.log"
Oct 31 02:17:29.180703 ip-10-0-139-174 kubenswrapper[4476]: E1031 02:17:29.180664    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vqn7z" containerName="extractor"
Oct 31 02:17:33.683991 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:33.683959    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5lczz_f90cbecc-43bb-41a8-b743-4cdcd12dce2f/dns/0.log"
Oct 31 02:17:34.096528 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:34.096493    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5lczz_f90cbecc-43bb-41a8-b743-4cdcd12dce2f/kube-rbac-proxy/0.log"
Oct 31 02:17:36.589556 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:36.589520    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gsv6r_7ae98732-67f4-420b-9e5b-e4e311ac9f9c/dns-node-resolver/0.log"
Oct 31 02:17:39.711398 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:39.711364    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64949dc4d4-2tx9z_634bbaef-6cd6-4686-8aed-f1797ef6c5bc/registry/0.log"
Oct 31 02:17:39.844818 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:39.844788    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:17:40.185616 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:40.185583    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7fcdb566-czqtv_4ca750c5-f8bb-4a8d-9579-71b6731da112/manager/0.log"
Oct 31 02:17:40.185807 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:40.185648    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7fcdb566-czqtv_4ca750c5-f8bb-4a8d-9579-71b6731da112/manager/1.log"
Oct 31 02:17:40.258648 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:40.258622    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lthm8_e4311d66-bb16-4a76-a021-527f4c1395f4/node-ca/0.log"
Oct 31 02:17:40.289293 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:40.289279    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7fcdb566-czqtv_4ca750c5-f8bb-4a8d-9579-71b6731da112/copy-manifests/0.log"
Oct 31 02:17:41.551894 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:41.551847    4476 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-174.us-west-1.compute.internal" podUID="7fc072a2ae8e85ec3464a94d2bac4401" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:17:42.992074 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:42.992040    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bb949dc9c-x6t7p_e766f760-98d7-44a0-978e-cf414a9e23b5/manager/0.log"
Oct 31 02:17:45.775558 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:45.775526    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kzt5t_9787be82-26ee-4d1c-ad0c-7d2de2f3e065/proxy/0.log"
Oct 31 02:17:46.278911 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:46.278877    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jww65_7b893def-2fc6-4350-a01f-6ee4969c48bd/konnectivity-agent/0.log" limit=1024
Oct 31 02:17:47.715226 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:47.715194    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58c8f9d5-2hkws_32e2d8b2-67ec-4f33-abe2-b5e27b1761a9/kube-auth-proxy/0.log"
Oct 31 02:17:47.831971 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:47.831945    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58c8f9d5-2hkws_32e2d8b2-67ec-4f33-abe2-b5e27b1761a9/kube-auth-proxy/0.log"
Oct 31 02:17:48.139338 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:48.139308    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-gbwwl_b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8/istio-proxy/0.log"
Oct 31 02:17:48.558391 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:48.558363    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dmjmv_a7ab6ca9-406f-471c-a4e4-1cc24dc493cb/payload-pre-processing/0.log"
Oct 31 02:17:48.662931 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:48.662899    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fc8c56547-gbwwl_b85531e9-78a0-4f6a-bbbd-6622c2d5c7a8/istio-proxy/0.log"
Oct 31 02:17:48.768495 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:48.768469    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vl9ww_c0881ec8-1f45-459a-b40c-29e2d2246ea0/payload-processing/0.log"
Oct 31 02:17:49.060663 ip-10-0-139-174 kubenswrapper[4476]: I1031 02:17:49.060635    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58c8f9d5-2hkws_32e2d8b2-67ec-4f33-abe2-b5e27b1761a9/kube-auth-proxy/0.log"